U xb@sddlZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z m Z m Z ddlmZmZmZddlTd.ddZd/d d Zd0d d Zd1ddZd2ddZd3ddZd4ddZd5ddZd6ddZd7dd Zedfd!d"Zd8d#d$Zd9d%d&Z d:d'd(Z!d;d*d+Z"dr?r@joindirnameabspathrArBrCrDrtimeseries_rmsrFrGrHrIrJrKrLrM) rNr6r5rOZrms_filerPrQZrms_listrRr$r$r%get_residual_rms]s2    rXmaskConnComp.h5cCsXt|}|d}|dkr,t|j|d}ntd|dSd|d<tj|||d|S)zDGenerate mask file for non-zero value of input multi-group hdf5 file FILE_TYPErrz5Only ifgramStack file is supported for now, input is Nmaskr-metadata)rrr nonzero_maskr<rwrite)Filer-ratrkr\r$r$r%r_s  r_ coherenceFTc Csdd}t|} | d} |s:ddt| dt| df}| dkrF|ntjtj|d} |dkrjd nd } | |rxd nd 7} | | d } || rtd||\}}||fSd tj|}d |}d |}d |}zt | d}| }| |r,zdd|Dd}Wnd }YnXn|}zdd|Dd}Wnd }YnX||kr||krt | ||gdddkrt| d|| \}}||fWSWn YnXd|krd}nd}| dkrt|}|j dd|j||||||d\}}|j}|j}| n| dkr>t|j||||d \}}ntj||d!d}|rtj|rtd"|tj|d#|d$d}tj||t|k<|dk rd%|||k<d|||k<t|}tj|g}|rtd&| t | d'}|||||t|}| dkrX|d(t|D].}|d)|||||||||fq&n4|d*t|D] }|d+||||fqj| t|d%kr|d}|d}||fS),aRead/Calculate Spatial Average of input file. If input file is text file, read it directly; If input file is data matrix file: If corresponding text file exists with the same mask file/AOI info, read it directly; Otherwise, calculate it from data file. Only non-nan pixel is considered. Parameters: File - string, path of input file maskFile - string, path of mask file, e.g. maskTempCoh.h5 box - 4-tuple defining the left, upper, right, and lower pixel coordinate saveList - bool, save (list of) mean value into text file reverseMask - bool, perform analysis within masked regions instead of outside of them threshold - float, calculate area ratio above threshold instead of spatial average Returns: meanList - list for float, average value in space for each epoch of input file dateList - list of string for date info date12_list, e.g. 101120-110220, for interferograms/coherence date8_list, e.g. 20101120, for timeseries file name, e.g. velocity.h5, for all the other file types Example: meanList = spatial_average('inputs/ifgramStack.h5')[0] meanList, date12_list = spatial_average('inputs/ifgramStack.h5', maskFile='maskTempCoh.h5', saveList=True) cSsPtj|tdt}dd|dddfD}dd|dddfD}||fS)Nr:cSsg|] }t|qSr$)r.0ir$r$r% sz;spatial_average..read_text_file..r cSsg|]}|qSr$r$rer$r$r%rhsr)rFrGrHrIrJ)fnameZ txtContentmeanListdateListr$r$r%read_text_filesz'spatial_average..read_text_filerZrr r rNZ SpatialAvgZ AreaRatioZRevMskz.txtz;Input file is spatial average txt already, read it directlyz# Data file: {} z# Mask file: {} z# AOI box: {} z# Threshold: {} rcSsg|]}d|kr|qS)z # AOI box:r$rer$r$r%rhsz#spatial_average..cSsg|]}d|kr|qS)z # Mask file:r$rer$r$r%rhsFr,skipz! already exists, read it directlyoffsetT print_msg)rr8r useMedian reverseMask thresholdr)r8rrtru)rzmask from file: r\rr z-write average value in space into text file: wz'# DATE12 Mean Btemp/days Bperp/m Num z%s %.4f %8.0f %8.1f %d z# DATE12 Mean z%s %.4f )rrrr>r?r@basenameendswithr<r=open readlinescloserArspatial_average pbaseIfgram tbaseIfgramrrrBrFnannanmeanr`lenrange) rarr8rZsaveListZcheckAoirtrurlrbrcprefixsuffixZtxtFilerjrkZ file_lineZ mask_lineZaoi_lineZ thres_linefllinesZ aoi_line_origZmask_line_origrsobjpbasetbasedatar\ZnumLinergr$r$r%r|s $                                 r|c Csvtj||d}|d}|dkrBtd|t|d}||fS|dkrtj|d}|s|dkr|d krtd }qd |krd }qd |}nF|dkr||krtj| dd |d}d|}n d |}|rtj |rt|d} | |fS|dkr:t |j |d} d |kr0d|d<d|d<n||d<n|dkrXt | } d|d<|rntj| ||d| |fS)aCalculate temporal average of multi-temporal dataset, equivalent to stacking For ifgramStakc/unwrapPhase, return average phase velocity Parameters: File : string, file to be averaged in time datasetName : string, dataset to be read from input file, for multiple datasets file - ifgramStack - only e.g.: coherence, unwrapPhase updateMode : bool outFile : string, output filename None for auto output filename False for do not save as output file Returns: dataMean : 2D array outFile : string, output file name Examples: avgPhaseVel = ut.temporal_average('ifgramStack.h5', datasetName='unwrapPhase')[0] ut.temporal_average('ifgramStack.h5', datasetName='coherence', outFile='avgSpatialCoh.h5', updateMode=True) r[rZ)rrzBWARNING: input file is not multi-temporal file: {}, return itself.rNr rrdzavgSpatialCoh.h5 unwrapPhasezavgPhaseVelocity.h5zavg{}.h5rzavgDisplacement{}.h5velocityzm/yearUNIT displacementr])rrr<r=rr>r?r@rwsplitrBrtemporal_averagerrr`) rarZ updateModer9rbrcrextZ processMarkZdataMeanr$r$r%r3sF          rc Cs|sgSt|tr|g}dd|D}g}tt|D]2}||}t|}|ttt|t|7}q6|r|dd|D}|dk rt|D]`}t |}|dkrd| kr| |q|dkrd| kr| |qd |} t | q|S) aRGet all existed files matching the input list of file pattern Parameters: file_list - string or list of string, input file/directory pattern abspath - bool, return absolute path or not coord - string, return files with specific coordinate type: geo or radar if none, skip the checking and return all files Returns: file_list_out - list of string, existed file path/name, [] if not existed Example: file_list = get_file_list(['*velocity*.h5','timeseries*.h5']) file_list = get_file_list('timeseries*.h5') cSsg|]}|dkr|qSNr$)rfxr$r$r%rhsz!get_file_list..cSsg|]}tj|qSr$)r>r?rVrer$r$r%rhsN)geor)ZradarrdrZrdcz"un-recognized input coord type: {}) isinstancerJrrglobsortedrMsetrrrremover= ValueError) file_listrVcoordZ file_list_outrgZfile0Z file_list0rirbrPr$r$r% get_file_listus0           rc s|sDdddddddg}dd d g}g}|D]|fd d |D7}q(g}z t|}Wn*|rxtd tdt|YdSXd}|D]H}t|}dD]4} z"tj|| ddd} |}WqWqYqXqq|s|rtddS|rtj|}|S)aFind lookup table file with/without input file pattern Parameters: filePattern - list of str abspath - bool, return absolute path or not print_msg - bool, printout message or not Returns: outFile - str, path of the lookup file zgeometryRadar.h5zgeometryGeo_tight.h5zgeometryGeo.h5zgeomap*lks_tight.transzgeomap*lks.transzsim*_tight.UTM_TO_RDCzsim*.UTM_TO_RDCinputsrmz ../inputscsg|]}tj|qSr$r>r?rT)rfrirUr$r%rhsz#get_lookup_file..z-ERROR: No geometry / lookup table file found!zIt should be like:N)r rangeCoordF)rrrrz.No lookup table info range/lat found in files.)rr<rrrr>r?rV) Z filePatternrVrrZfileListZdirListZ existFilesr9rirbdsNamedsetr$rr%get_lookup_filesN      rrcst|tr|g}|D]}|tkrtd|qs.)z *geometry*.h5z*/*geometry*.h5z../*/geometry*.h5)rrzNo geometry file found.Nc3s|]}|tkVqdSr)rget_dataset_list)rfr)rir$r% sz$get_geometry_file..z&No geometry file with dataset {} found)rrJrrr=r>getcwdrrr<rManyrr?rV)Z dset_listrrrVrrr fname_listrr$)rirr%get_geometry_files0       r=cCs|d}t|}|D]$\}}||kr|||krd}q|sPtd||S|d}t|d}t|dD]} dd| |d D} | d sXt | d krX| d }t | d d d dd }||krX|||krX|} dD]} | | d | } q|d| } t | | d } | |||}| | |d } td |||||| ql|t|||S)zEUpdate option value in template_file with value from input extra_dictFTz)No new option value found, skip updating z.tmprvrncSsg|] }|qSr$)striprer$r$r%rhsz(update_template_file..r )%#r rmr)r []()z\{}z[\s]*z {}: {} --> {})r read_templateitemsrr<ryrr startswithrrJreplacer=refindallr`r{shutilmove)Z template_fileZ extra_dict delimiterupdateZ orig_dictkeyvalueZtmp_fileZf_tmplinecZ value2searchsymbolZ old_value_strZ new_value_strr$r$r%update_template_files8       rcCst|}t||}|s$td|St|d}t|D]p\}}|dksT|dkrz"|j ||rttd |WqYqXqr?rBrer$r$r%rszrun_or_skip..rr z'{} exists, but can not read, remove it.cSsg|]}tj|qSr$r>r?getmtimerer$r$r%rhszrun_or_skip..cSsg|]}tj|qSr$rrer$r$r%rhsz({} exists and is newer than {} --> skip.ro) rrJallrrr<r=r>rrmaxmin)r-r.r/rrrbwidthZt_inZt_outr$r$r%rAs2  rA"defaults/smallbaselineApp_auto.cfgc Cstjtjtj|}t|}d}||d }|dkrF||}|dkrVd|d<| D](\}}|dkr^|| kr^||||<q^dddddd }| D](\}}| }|| kr||||<q|S) z7Replace auto value based on the input auto config file.zmintpy.compute.clusterautolocalZ40zmintpy.compute.numWorkerTFN)yestruer)falsenone) r>r?rTrUmintpy__file__rrgetlowerrr) Z templateDictZ auto_fileZtemplateAutoFileZtemplateAutoDictZ cluster_keyZclusterrrZ specialValuesr$r$r%check_template_auto_values,  rc Csjt}t|}|d}t|d} t|d} td|||sbtj|\} } d| | }|dkrn|}tj |rt |d} td|nt j | | ft j d } td d }|r6tjtj|d} tjtj| d | }t|d 6}|d||d||d|W5QRX|dkr@tj||ddtdt|}t|}tj|d}t|D]}|rt|d}|d||W5QRXtj |||dd}t|| |||dd}tj||d||dd| d| gdd|j|dd|d|dq|td|n|dkrt|}|jdd |shd!}t !|d8}||}d"|}||"kr||}td#|n,|j#||j$| | ft j%dd d$}td%|tj|j$d}t|j$D]}|r.t|d }|dt&|j'|W5QRX||d d d d f}t|| |||dd}|||d d d d f<|j|dd|d|j$dq|td|W5QRXn|rt|d}|d|dW5QRX|s|d&krd&}tj ||dd}t|| |||dd}td'|tj|||d(t(t|d)\}}td*|||S)+a Remove ramp from each 2D matrix of input file Parameters: fname : str, data file to be derampped ramp_type : str, name of ramp to be estimated. mask_file : str, file of mask of pixels used for ramp estimation out_file : str, output file name datasetName : str, output dataset name, for ifgramStack file type only save_ramp_coeff : bool, save the estimated ramp coefficients to text file Returns: out_file : str, output file name rZr r zremove {} ramp from file: {}z {}_ramp{}rrzread mask file: r:zuse mask of the whole areaNzrampCoeff_{}.txtrvz# input file: {} z# output file: {} z# ramp type: {} rT)ref_filerrz,estimating phase ramp one date at a time ...)maxValueaz{} r[)r5r^ coeff_filer F)rblockrrz{}/{})rzfinished writing to file: {}rqrz{}_rampzaccess HDF5 dataset /{})shaper;chunks compressionzcreate HDF5 dataset /{}rzwriting >>> {})r-r<z'time used: {:02.0f} mins {:02.1f} secs.))timerrrr<r=r>r?r@rBrrFonesbool_rwrTrUryr`r layout_hdf5r get_date_listrr progressBarrrwrite_hdf5_blockrr{rrrarcreate_dataset numIfgramrKrJ date12Listdivmod)rir5r6r-rZsave_ramp_coeff start_timerbrclengthrfbasefextr\rrrRnum_dateprog_barrgrrdsZ dsNameOutZdsOutmsr$r$r%rDs            "     $$ rD)N)r'r()r'r()rYN)rdNNFTFN)rdFN)FN)NFT)NrTT)r)NN)T)NTT)r)NNNF)$r>rrrrrnumpyrFrmintpy.objectsrrrr mintpy.utilsrrrmintpy.utils.utils0r&rSrXr_r|rrrrrdictrrrrArrDr$r$r$r% s@  - 3   B * 5 " .# +  9 #