Ytdyklly

I have 10GB files in which i want to count the occurrences of some specific text i.e TY[0-9].

Example file:

ABC,2A,2018-07-06,2018-06-20 00:00:00

BCD,TY1,2018-07-06,2018-06-20 00:00:00

EFG,TY2,2018-07-06,2018-06-20 00:00:00

IGH,2A,2018-07-06,2018-06-20 00:00:00

I want to get the count of all text starting with TY and then a digit. I tried using egrep but am not getting the correct result.

egrep  "^TY[0-9]" Filename

The main issue with your attempted solution is that it assumes that the sting TY occurs at the start of the line (you are anchoring the expression there with ^), but it doesn't. It occurs at the start of the second comma-delimited field.

Using awk to count the number of times the second comma-delimited field in the file starts with the string TY followed by a digit:

awk -F, '$2 ~ /^TY[[:digit:]]/ { n++ } END { print n }' filename

I'm wondering whether using cut in combination with grep would be quick? Cutting out the second column would give grep less data to work with, and so it may be quicker than just grep alone.

cut -d, -f2 filename | grep -c '^TY[[:digit:]]'

... but I'm not sure.

After some testing on my OpenBSD system, using a 1.1GB file, the cut+grep is actually almost 50% quicker than awk (8 seconds vs. 15 seconds). And a pure grep solution (grep -Ec '<TY[0-9]' filename, taken from glenn's solution) takes 13 seconds.

So if the string is to picked out of the second field only, one may gain some time by extracting only that field before matching.

You want to use a word boundary instead of the start-of-line anchor:

$ grep -Ec '<TY[0-9]' file

2

Note: that is a count of all lines with a "TY word". It is not a count of all "TY word"s. If you can have more than one per line, then

$ grep -Eo '<TY[0-9]' file | wc -l

If you want to find the number of occurrence of a , delimited field that starts with TY and is followed by any number of decimal digits, you could do:

<file perl -lne '$n += () = /(?<![^,])TYd+(?![^,])/g; END{print 0+$n}'

Which on an input like:

TY1,TY2,TY,TYFOO

TY213,X-TY2,TY4

Would return 4 (TY1, TY2, TY213, TY4).

(?<!...) and (?!...) are respectively negative look behing and ahead operators. So here, we're looking for TY followed by one or more (+) digits (d), provided its neither preceded nor followed by a character other than ,.

Another way to do it would be to convert ,s to newlines and count the number of resulting lines that start with TY followed by one or more digits:

<file tr , 'n' | LC_ALL=C grep -xEc 'TY[[:digit:]]+'

(on my system, that's about 10 times as fast as the perl solution)