G. Lang 11/2016 1) GLSEA + ice files Alex Assuncao produced for 1995-2007. 2069 GLSEA-ICE : ls -l /mnt/atoz/cwdata/p/m/lcb/glsea_with_ice_1995-2007 total 933120 -rwxrwxr-x 1 cw 1023432400 32735789 Nov 4 10:07 1995.zip -rwxrwxr-x 1 cw 1023432400 29365076 Nov 4 10:07 1996.zip -rwxrwxr-x 1 cw 1023432400 31641582 Nov 4 10:07 1997.zip -rwxrwxr-x 1 cw 1023432400 45162658 Nov 4 10:07 1998.zip -rwxrwxr-x 1 cw 1023432400 43061112 Nov 4 10:07 1999.zip -rwxrwxr-x 1 cw 1023432400 43660658 Nov 4 10:07 2000.zip -rwxrwxr-x 1 cw 1023432400 36226507 Nov 4 10:07 2001.zip -rwxrwxr-x 1 cw 1023432400 44581101 Nov 4 10:07 2002.zip -rwxrwxr-x 1 cw 1023432400 33096969 Nov 4 10:07 2003.zip -rwxrwxr-x 1 cw 1023432400 34390449 Nov 4 10:07 2004.zip -rwxrwxr-x 1 cw 1023432400 32640753 Nov 4 10:07 2005.zip -rwxrwxr-x 1 cw 1023432400 36944408 Nov 4 10:07 2006.zip -rwxrwxr-x 1 cw 1023432400 33165753 Nov 4 10:07 2007.zip 2) unzip and place *.asc files in YYYY.asc dirs. These YYYY.asc files are for ice year starting on Dec 1 of previous year. 3) run convert_asc_i_to_dat.pro to produce *glsea_i.dat files from YYYY.asc/*.asc files. *.asc files start with YYYY-101.asc. This is Dec 1, of YYYY-1. This is either DOY 335, or 336 if YYYY-1 is leap year. The *glsea_i.dat files are placed in their calendar-year YYYY dirs. So, each YYYY dir has *glsea_i.dat files from Jan 1 to May 31, and files from Dec 1 to Dec 31. 4) run convert_asc_to_dat.pro to produce *glsea.dat files from coastwatch *glsea.asc file (on atoz), since coastwatch didn't have *glsea.dat files for these years. Only *glsea.asc. 5) So now the YYYY dirs are populated with *glsea.dat and *glsea_i.dat 6) run create_glsea_ice_nc.pro to create annual nc files, YYYY_glsea_ice.nc These files contain GLSEA surface water temps (>0 C), negative values indicate ice fraction (0-1). 7) run reduce_size.sh to convert from classic to netCDF-4 classic model, and compress (10X smaller!) 8) mv YYYY_glsea_ice.nc to /mnt/archive/lang/GLSEA/glsea_ice_nc_files/ 2207 2007 : ls -l /mnt/archive/lang/GLSEA/glsea_ice_nc_files/ total 5566720 total 5563648 -rw-r--r-- 1 lang users 145399107 Nov 16 19:36 1995_glsea_ice.nc -rw-r--r-- 1 lang users 128033005 Nov 16 19:36 1996_glsea_ice.nc -rw-r--r-- 1 lang users 129858559 Nov 16 19:37 1997_glsea_ice.nc -rw-r--r-- 1 lang users 145644069 Nov 16 19:37 1998_glsea_ice.nc -rw-r--r-- 1 lang users 143716449 Nov 16 19:37 1999_glsea_ice.nc -rw-r--r-- 1 lang users 142273228 Nov 17 07:52 2000_glsea_ice.nc -rw-r--r-- 1 lang users 137493979 Nov 16 19:37 2001_glsea_ice.nc -rw-r--r-- 1 lang users 147188804 Nov 16 19:37 2002_glsea_ice.nc -rw-r--r-- 1 lang users 126024023 Nov 16 19:37 2003_glsea_ice.nc -rw-r--r-- 1 lang users 125960897 Nov 16 19:37 2004_glsea_ice.nc -rw-r--r-- 1 lang users 121911453 Nov 16 19:37 2005_glsea_ice.nc -rw-r--r-- 1 lang users 128623428 Nov 16 19:37 2006_glsea_ice.nc -rw-r--r-- 1 lang users 123032121 Nov 17 07:54 2007_glsea_ice.nc -rw-r--r-- 1 lang users 119089383 Nov 17 08:54 2008_glsea_ice.nc -rw-r--r-- 1 lang users 112439305 Nov 17 08:55 2009_glsea_ice.nc -rw-r--r-- 1 lang users 120892789 Nov 16 19:37 2010_glsea_ice.nc -rw-r--r-- 1 lang users 131478895 Nov 16 19:37 2011_glsea_ice.nc -rw-r--r-- 1 lang users 133666153 Nov 16 19:37 2012_glsea_ice.nc -rw-r--r-- 1 lang users 134425734 Nov 16 19:38 2013_glsea_ice.nc -rw-r--r-- 1 lang users 117754136 Nov 16 19:38 2014_glsea_ice.nc -rw-r--r-- 1 lang users 121944151 Nov 16 19:38 2015_glsea_ice.nc -rw-r--r-- 1 lang users 110222968 Nov 16 19:38 2016_glsea_ice_001-320.nc 22 nc files, about 2.85 GB total Summary: 1995-2003: used *glsea_i.dat created by Alex Assuncao and Coastwatch *glsea.dat (converted from *glsea.asc) to create the *_glsea_ice.nc 2004-2007: used *glsea_i.dat created by Alex Assuncao and Coastwatch *glsea.dat to create the *_glsea_ice.nc 2008-2016: used Coastwatch *glsea_i.dat and Coastwatch *glsea.dat to create the *_glsea_ice.nc *** Problems Encountered and Fixes *** 0) Alex's asc 1995 ascii files stated on Jan 1, 1995, not Dec 1, 1994 as the rest of the years did. 1) 2001.asc/2001-110.asc contained bad characters and could not be processed. This is 2000 DOY 345. wrote idl program, fix.pro to: use SST from 2000_345_glsea.dat and ice from 2000_344_glsea_i.dat to create 2000_345_glsea_i.dat 2) wrote and run check_size.sh to count the number of water cells in the *glsea_i.dat created by convert_asc_i_to_dat.pro These resulting dat files had 1 or 2 extra water cells. Should have only 148682. A problem with Alex's 2007 *.asc files. 2007/2007_015_glsea_i.dat 148683 2007/2007_029_glsea_i.dat 148684 2007/2007_036_glsea_i.dat 148684 2007/2007_043_glsea_i.dat 148684 2007/2007_050_glsea_i.dat 148684 2007/2007_057_glsea_i.dat 148684 2007/2007_064_glsea_i.dat 148684 2007/2007_071_glsea_i.dat 148684 2007/2007_078_glsea_i.dat 148684 2007/2007_085_glsea_i.dat 148684 2007/2007_092_glsea_i.dat 148684 2007/2007_099_glsea_i.dat 148684 2007/2007_106_glsea_i.dat 148684 So I used fix.pro to: use SST from 2007_015_glsea.dat and ice from 2007_014_glsea_i.dat to create 2007_015_glsea_i.dat use SST from 2007_029_glsea.dat and ice from 2007_028_glsea_i.dat to create 2007_029_glsea_i.dat use SST from 2007_036_glsea.dat and ice from 2007_035_glsea_i.dat to create 2007_036_glsea_i.dat use SST from 2007_043_glsea.dat and ice from 2007_042_glsea_i.dat to create 2007_043_glsea_i.dat use SST from 2007_050_glsea.dat and ice from 2007_049_glsea_i.dat to create 2007_050_glsea_i.dat use SST from 2007_057_glsea.dat and ice from 2007_056_glsea_i.dat to create 2007_057_glsea_i.dat use SST from 2007_064_glsea.dat and ice from 2007_063_glsea_i.dat to create 2007_064_glsea_i.dat use SST from 2007_071_glsea.dat and ice from 2007_070_glsea_i.dat to create 2007_071_glsea_i.dat use SST from 2007_078_glsea.dat and ice from 2007_077_glsea_i.dat to create 2007_078_glsea_i.dat use SST from 2007_085_glsea.dat and ice from 2007_084_glsea_i.dat to create 2007_085_glsea_i.dat use SST from 2007_092_glsea.dat and ice from 2007_091_glsea_i.dat to create 2007_092_glsea_i.dat use SST from 2007_099_glsea.dat and ice from 2007_098_glsea_i.dat to create 2007_099_glsea_i.dat use SST from 2007_106_glsea.dat and ice from 2007_105_glsea_i.dat to create 2007_106_glsea_i.dat 3) Missing Coastwatch *glsea_i.dat (on atoz) 2008: 005,006,007,008,009,011 2008: 015,016,017 2008: 358,359 2008: 362 So I used fix.pro to: use SST from 2008_005_glsea.dat and ice from 2008_004_glsea_i.dat to create 2008_005_glsea_i.dat use SST from 2008_006_glsea.dat and ice from 2008_004_glsea_i.dat to create 2008_006_glsea_i.dat use SST from 2008_007_glsea.dat and ice from 2008_004_glsea_i.dat to create 2008_007_glsea_i.dat use SST from 2008_008_glsea.dat and ice from 2008_004_glsea_i.dat to create 2008_008_glsea_i.dat use SST from 2008_009_glsea.dat and ice from 2008_004_glsea_i.dat to create 2008_009_glsea_i.dat use SST from 2008_010_glsea.dat and ice from 2008_004_glsea_i.dat to create 2008_010_glsea_i.dat use SST from 2008_015_glsea.dat and ice from 2008_014_glsea_i.dat to create 2008_015_glsea_i.dat use SST from 2008_016_glsea.dat and ice from 2008_014_glsea_i.dat to create 2008_016_glsea_i.dat use SST from 2008_017_glsea.dat and ice from 2008_014_glsea_i.dat to create 2008_017_glsea_i.dat use SST from 2008_358_glsea.dat and ice from 2008_357_glsea_i.dat to create 2008_358_glsea_i.dat use SST from 2008_359_glsea.dat and ice from 2008_357_glsea_i.dat to create 2008_359_glsea_i.dat use SST from 2008_363_glsea.dat and ice from 2008_362_glsea_i.dat to create 2008_363_glsea_i.dat 2009: 336,337,338,339,340,341 So I used fix.pro to: use SST from 2009_336_glsea.dat and ice from 2009_335_glsea_i.dat to create 2009_336_glsea_i.dat use SST from 2009_337_glsea.dat and ice from 2009_335_glsea_i.dat to create 2009_337_glsea_i.dat use SST from 2009_338_glsea.dat and ice from 2009_335_glsea_i.dat to create 2009_338_glsea_i.dat use SST from 2009_339_glsea.dat and ice from 2009_335_glsea_i.dat to create 2009_339_glsea_i.dat use SST from 2009_340_glsea.dat and ice from 2009_335_glsea_i.dat to create 2009_340_glsea_i.dat use SST from 2009_341_glsea.dat and ice from 2009_335_glsea_i.dat to create 2009_341_glsea_i.dat 2015: 159,160,161,162,163,164,165,166,167 No further action, as 2015 158 contained no ice. So assume no ice for 159-167. 4) I fixed and regenerated 1995_glsea_ice.nc. I manually edited the last 3 days of 2005 and the first 17 days of 2006 (*.asc) and created corresponding *.dat files. There was a problem at the tip of the Keweenaw Peninsula. There are spurious high temps for many cells. It's not as simple as finding/removing values > say 35, because during the creation of these asc files, the high values got spread trough some sort of interpolation or smoothing to nearby cells. So even values as low as 3 needed to be fixed. This cannot be done with a program. I needed to manually edit and 'decide' which cells need fixing. Songzhi has placed these on the cwdata server. Anne recreated Alex's 2006.asc files. I regenerated the 2005 and 2006 nc files, as well as the 2016 (complete year) nc file.