Home/ Questions/Q 6906491
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T08:19:01+00:00

I have timeseries data for many terms an example of which is below: term1

  • 0

I have timeseries data for many terms an example of which is below:

term1 = [0.0, 0.0, 0.0, 0.0, 2.2384935833581433e-06, 3.938767914008819e-06, 0.0, 0.0, 1.1961851263949013e-06, 0.0, 2.278384397623645e-06, 1.100158422812885e-06, 0.0, 1.095521835393462e-06, 0.0, 0.0, 1.6933152148605343e-06, 0.0, 8.460737945563612e-07, 8.949410770794851e-07, 0.0, 2.8698467119209605e-06, 0.0, 0.0, 0.0, 3.9163008188985015e-06, 2.2244961516216576e-06, 0.0, 0.0, 1.9407903674692482e-06, 0.0, 0.0, 0.0, 0.0, 9.514657329616274e-07, 1.94463053478312e-06, 0.0, 0.0, 0.0, 2.0373216961518047e-06, 1.8835690620014428e-06, 0.0, 0.0, 0.0, 0.0, 9.707946148081127e-07, 0.0, 0.0, 1.6121985390256838e-06, 1.9547361301697883e-06, 0.0, 2.2876018840689116e-06, 2.208826914114183e-06, 1.9640500282823203e-06, 0.0, 2.6234669115235785e-06, 0.0, 0.0, 0.0, 1.986207773222741e-06, 1.049193537387487e-06, 1.090723073046815e-06, 0.0, 1.0257546476943088e-06, 9.179053033814713e-07, 0.0, 0.0, 0.0, 0.0, 9.335621182897889e-07, 0.0, 0.0, 0.0, 0.0, 2.1267500494469387e-06, 2.215050381320923e-06, 2.163720040591388e-06, 1.937729136470388e-06, 1.6037643556956889e-06, 1.313906783569333e-06, 0.0, 1.0064645216223805e-06, 1.876346865234201e-06, 9.504447606257348e-07, 2.017974095266539e-06, 0.0, 2.120782823355757e-06, 0.0, 0.0, 0.0, 0.0, 9.216394491176685e-07, 0.0, 0.0, 1.0401357169083422e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0089962853658684e-06, 1.8249773702806084e-06, 0.0, 1.2890950295073852e-06, 5.42812725267281e-06, 1.9185480428411778e-06, 2.6955316172381044e-06, 0.0, 0.0, 1.0070239923466176e-06, 0.0, 1.021152145542773e-06, 9.919749228739498e-07, 1.9293082175989564e-06, 9.802489636317832e-07, 1.0483850676418046e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 1.9369409504181854e-06, 0.0, 4.619620451983665e-06, 0.0, 6.0795324434248845e-06, 0.0, 1.5312669396405198e-06, 1.2797051559320733e-06, 1.1002903666277531e-06, 0.0, 1.0054768323055684e-06, 2.060260561153169e-06, 1.0898719291496056e-06, 3.4605907920600203e-06, 3.3500051925080486e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.5521496510980315e-06, 0.0, 0.0, 0.0, 3.01862187836765e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.849817053093449e-06, 6.5552277941658475e-06, 1.985771944021089e-06, 1.010233667047188e-06, 9.802307070992228e-07, 5.605931075077432e-06, 3.651067480854715e-06, 0.0, 0.0, 2.9476960807432912e-06, 1.834478659509754e-06, 0.0, 0.0, 0.0, 0.0, 3.3801712394749917e-06, 0.0, 2.2884970981856794e-06, 1.02014792144861e-06, 2.906143199237428e-06, 9.807873564740302e-07, 0.0, 2.106593638087213e-06, 3.0329622335542676e-06, 2.9093758515985565e-06, 0.0, 2.12762335960239e-06, 9.614820669172289e-07, 9.264114341404848e-07, 0.0, 0.0, 9.073611487918033e-07, 0.0, 0.0, 0.0, 6.0360958532021484e-06, 0.0, 4.553288270957079e-06, 2.0712553257152562e-06, 3.292603824030081e-06, 2.690786880261329e-06, 2.301011409565074e-06, 2.029661472762958e-06, 0.0, 9.657114492818003e-07, 9.948942029504583e-07, 1.028682761437152e-06, 2.0694207898151387e-06, 3.845369982272845e-06, 9.048250701691842e-07, 1.7726379156614332e-06, 0.0, 9.238711680133629e-07, 9.231112912203808e-07, 9.422814896339613e-07, 0.0, 1.2123519263665934e-06, 0.0, 0.0, 2.1675188628329036e-06, 0.0, 4.498718989767663e-06, 0.0, 0.0, 2.650273839544471e-06, 1.1954029583832415e-06, 4.180999656112778e-06, 1.9036523473937095e-06, 9.75877289286136e-07, 0.0, 2.093618232902467e-06, 1.032899928523325e-06, 0.0, 4.473312219299659e-06, 8.762705923589204e-07, 0.0, 0.0, 1.792797436299666e-06, 0.0, 0.0, 1.1974513445582422e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.1404264054329915e-06, 3.061324451410658e-06, 9.84829683554526e-07, 2.932895354293759e-06, 2.0897069394988045e-06, 0.0, 2.128187093183736e-06, 0.0, 4.686861415132188e-06, 6.37755683086446e-06, 1.8420463661490824e-06, 2.8347403094402523e-06, 1.9033842171380715e-06, 6.909144746582441e-06, 0.0, 0.0, 1.5479612576256442e-06, 5.621978186724636e-06, 2.087185930697078e-06, 1.3168406359462377e-05, 1.9676130885622652e-05, 1.9988766313331908e-05, 3.1801079228204546e-05, 3.322824899588385e-05, 2.0358501231090545e-05, 1.2383952049337664e-05, 1.8052256532066507e-05, 7.770543617518302e-06, 9.226179797741636e-06, 4.400430362089412e-06, 4.333084180992927e-06, 7.477274426653279e-06, 3.0526428255261993e-06, 4.952368123389242e-06, 1.2578584707962998e-05, 0.0, 2.121750274236223e-06, 0.0, 2.38940918273843e-06, 0.0, 1.5693511988273807e-06, 0.0, 0.0, 4.520448247648237e-06, 4.0303440122522456e-05, 2.8660979509446863e-05, 2.4793768971660722e-05, 3.957070185234852e-05, 2.64488881248099e-05, 6.428381095168035e-05, 5.6557662521419976e-05, 6.855540059858658e-05, 7.079288025889968e-05, 7.135683422742382e-05, 5.5663480860112103e-05, 8.088436527379357e-05, 7.142268494354861e-05, 8.243171356847987e-05, 7.658173644233611e-05, 5.4275733753644613e-05, 2.7329513031804995e-05, 1.8666856995404658e-05, 2.5061514626811264e-05, 9.707359513272993e-06, 2.233654188450612e-05, 2.0577084330035857e-05, 6.037067595033506e-05, 5.358585847760433e-05, 6.353114888415205e-05, 4.913406130358561e-05, 6.253876100291326e-05, 5.783647108547192e-05, 5.29265883017118e-05, 4.295770587763158e-05, 0.00012513639867455526, 0.0001264425725280477, 0.00010075697417828198, 7.700585441944497e-05, 6.390017630639553e-05, 6.862379380485504e-05, 8.118867124374998e-05, 8.928305705187346e-05, 8.923668314113125e-05, 5.0862818355003976e-05, 2.5192448399293734e-05, 1.9491995287268695e-05, 1.1397180337584482e-05, 1.8548131739430545e-05, 2.8274146120787152e-05, 2.9861740143137274e-05, 5.749201435920551e-05, 8.676081065218611e-05, 0.00011692016691003383, 6.18107213073443e-05, 8.31986307882476e-05, 5.661490072734421e-05, 6.637785526376392e-05, 6.189842468509176e-05, 5.077848495281155e-05, 3.7630726455798414e-05, 6.325167842846687e-05, 7.447442335517917e-05, 7.881778491014126e-05, 8.347575938861497e-05, 6.553610066345062e-05, 6.209221186256924e-05, 4.671174184109858e-05, 4.583301504850661e-05, 2.9423292949863758e-05, 1.9969520206001368e-05, 1.3386836054765546e-05, 1.0233804045678584e-05, 2.3371876153986385e-05, 3.701784260013326e-05, 2.6804842191646374e-05, 3.729558727386808e-05, 7.011179438698544e-05, 4.616049584765358e-05, 6.019787395273405e-05, 8.312188292939014e-05, 6.281596430043117e-05, 6.370630077282333e-05, 6.169767733530766e-05, 6.099512039036877e-05, 7.192322709245217e-05, 6.727547574464268e-05, 4.891125624919348e-05, 8.775231227342841e-05, 9.349358010749929e-05, 4.85363097385816e-05, 4.475820776946539e-05, 1.9528637281926147e-05, 1.5243002033035396e-05, 1.4322461630125293e-05, 1.0492122514416176e-05, 1.1956759574674148e-05, 1.5232250274180506e-05, 3.394641638997643e-05, 2.6115894879792267e-05, 4.868559048521277e-05, 5.612535494090208e-05, 3.269545148571978e-05, 4.967751016319062e-05, 4.8382804751191425e-05, 5.1860846075881435e-05, 4.4034258653232213e-05, 5.362193446127224e-05, 6.213052893181175e-05, 8.561827093901839e-05, 5.877682625663455e-05, 0.0, 0.0, 7.105805443046969e-05, 0.0, 0.0, 2.31393994554528e-05, 7.05044594070575e-06, 2.21491300929156e-05, 4.926848615186025e-06, 1.0752514744385843e-05, 1.4745260873155369e-05, 1.976297604068538e-05, 3.094705732168692e-05, 5.068338091939653e-05, 2.655137469742496e-05, 3.0142790705685793e-05, 3.89279249469607e-05, 6.264176821226988e-05, 3.598536226187379e-05, 4.430195278344506e-05, 2.7501831818440764e-05, 1.7243328268903956e-05, 1.2049184772240285e-05, 2.1016880758625327e-05, 3.411070201956675e-05, 3.1893789428697184e-05, 1.8509911029027654e-05, 3.920735117199027e-05, 3.700840501998454e-05, 8.529330234343347e-06, 1.1007881643256571e-05, 4.661265813344272e-06, 7.306007242688513e-06, 2.6772256446090046e-06, 3.0075821145106816e-06, 6.713527085725027e-06, 2.204123915846549e-05, 7.880065404542858e-06, 4.3539870647002475e-05, 6.0898558226633984e-05, 7.956054903697144e-05, 4.80968670903199e-05, 3.476307626484116e-05, 3.233622280581405e-05, 4.097520999795124e-05, 1.6048981491512094e-05, 3.4725910431663494e-05, 2.3840743831207534e-05, 4.194630872483221e-05, 3.472531193096608e-05, 2.9240209403218155e-05, 2.5871727972711297e-05, 1.1918039641386187e-05, 1.2189485552920143e-05, 8.254477280067191e-06, 5.343416003103456e-06, 0.0, 4.795714549478586e-06, 6.705621859254362e-06, 9.484831383410081e-06, 2.503719812292549e-05, 1.9037212038371403e-05, 2.448114104715256e-05, 3.2063674685728836e-05, 2.73499598297465e-05, 2.6255716088190032e-05, 2.930473870366029e-05, 2.490020970307041e-05, 2.4037259675477766e-05, 1.8683888229243836e-05, 9.573344744760269e-06, 2.01589736663327e-05, 2.8955116521484698e-05, 1.934869527601605e-05, 2.1111566182648825e-05, 1.0035410663340645e-05, 4.154485944681635e-06, 8.468739061212046e-06, 8.415088253238056e-06, 1.3883239181832948e-06, 0.0, 2.9995080806747692e-06, 1.6303266848611124e-06, 3.714448088730736e-06, 8.976418947425114e-06, 9.729566693747293e-06, 3.3588780313874236e-05, 1.7154466165266127e-05, 1.9646193877372823e-05, 9.475852684603824e-06, 9.763432041631274e-06, 2.5840349706066022e-05, 1.4272109443725072e-05, 2.262309793162043e-05, 1.733067926359468e-05, 8.405046389852468e-06, 1.6489619195801272e-05, 6.6721749177376435e-06, 5.2645543870584616e-06, 5.563468043439559e-06, 5.668953522517651e-06, 2.564151874715539e-06, 4.72535638047152e-06, 1.1322053548784465e-06, 4.683593955822e-06, 5.170243182388084e-06, 1.4458242427134072e-06, 5.110793484760465e-06, 8.06295555698897e-06, 1.7613618850094893e-05, 1.3702227753862316e-05, 1.2582942563061514e-05, 1.5863866870429223e-05, 5.763738591399926e-06, 5.010013765012819e-06, 3.355941190486578e-06, 1.2264709219075303e-05, 3.0533139142568385e-06, 5.2266756983622735e-06, 3.0845411025383717e-06, 7.013177761012944e-06, 1.5042033081191253e-05, 7.918060391926394e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0146814988996414e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 1.4253135689851767e-05, 0.0, 1.4218885523752648e-05, 0.0, 1.539361472861057e-05, 0.0, 3.981789947307646e-05, 0.0, 1.2433017120264575e-05, 1.2777756481516975e-05, 1.2764382267720154e-05, 0.0, 1.2131652695046646e-05, 0.0, 0.0, 1.9324418335008117e-05, 0.0, 5.3279343598486864e-05, 1.316118503310038e-05, 8.202637968370628e-06, 8.606938339893733e-06, 2.2898281255009e-06, 0.0, 3.510274573677153e-06, 1.6317872149471709e-06, 4.578600840631114e-06, 3.877291479264245e-06, 2.8741881616021876e-06, 0.0, 1.0729671296519832e-05, 4.808871405969733e-06, 4.534612698729401e-06, 4.3333188889370365e-06, 0.0, 2.743032696949748e-06, 4.019804235533729e-06, 1.8078426019917002e-06, 5.444991968636846e-06]

Each element is the combined signal for an hour and the list is over 24 days. Therefore, should have 24 days * 24 hours = 576 elements.

I am interested in the dynamics of the signal of each term over the course of days. However, confounding to this is the baseline changes of the signal within a day. I also have time series for basic terms that capture this baseline signal during a day such as the following.

baseline = [0.0056738537419516195, 0.005420397434626666, 0.005019676698052322, 0.004214006968007205, 0.004143451622795924, 0.00373395198248036, 0.0037080495988714344, 0.0036409523281401525, 0.003919898659196092, 0.004388163261294729, 0.004595501330006892, 0.005097033972892097, 0.0052221285817481335, 0.005184009325081863, 0.005273633551787361, 0.005053393415305126, 0.004444952439008902, 0.004552838940992971, 0.004808237374463801, 0.004895327691624783, 0.005059086256629757, 0.005598114319387153, 0.005952632334681949, 0.005717805004263755, 0.006126432469142252, 0.005592477569387059, 0.004920585487387107, 0.004318038669070883, 0.003877225571288378, 0.0036583898426795327, 0.0037336953886437474, 0.0037760782770061294, 0.0042338376814351954, 0.004192003050341723, 0.0046450557083186645, 0.004900468947653463, 0.005272546953959605, 0.005265723105151999, 0.0052304537716869855, 0.005121826744125637, 0.005224078793461002, 0.00501027352884918, 0.004871995260153345, 0.004863044486978714, 0.005310347911635811, 0.0058606870895965765, 0.00596470801561322, 0.005997180909289017, 0.00588291246890472, 0.005328610690842843, 0.004941976393965633, 0.004426509645673344, 0.0041172533679088375, 0.0038888190559989945, 0.003785501144341545, 0.0038683019610415165, 0.003826474222198437, 0.004178336738982966, 0.004574137078717032, 0.004854291797756379, 0.005216590267890586, 0.00514712218170792, 0.005217487414098377, 0.005239554740422529, 0.005138433888329476, 0.0050591314342241745, 0.005099277335119803, 0.00469742744667216, 0.005140145739820509, 0.005534156237221868, 0.006098190503066302, 0.00627542293362276, 0.005859315099582288, 0.0055863100804189264, 0.005193523620749424, 0.004680401455731111, 0.0041370327176107335, 0.003790198190936078, 0.0037143182477912154, 0.0037406926128218908, 0.0038838040372017974, 0.00413455625482474, 0.004309030576010342, 0.004768381364059312, 0.004905695025592956, 0.0050965947056771715, 0.005178951654634759, 0.005250840996574289, 0.005083679897873849, 0.0050438189257025106, 0.00465730975130931, 0.004511425103430987, 0.004631293617276186, 0.0049417738509291015, 0.005495036992426772, 0.0056409591251836465, 0.005487421237451456, 0.005093572532544252, 0.005043698439924855, 0.004837771685295603, 0.0038251289273366134, 0.003817852658627033, 0.003792612420533331, 0.003922716174790973, 0.00412646233748187, 0.004534488299255124, 0.004712687777471286, 0.005096266809417225, 0.00523394116440575, 0.005257672264041691, 0.005305086574696615, 0.005100654966986151, 0.004965826463906992, 0.005073115958176456, 0.00469441228683261, 0.004553136348768357, 0.004723653823501124, 0.004726168081415059, 0.005290955031631742, 0.005325956759690025, 0.005453676000994151, 0.005531903354394338, 0.005121462750913455, 0.004790546408707061, 0.004460326025284444, 0.003982093750443299, 0.0036151869988949132, 0.0035295702958713982, 0.003722662298606401, 0.004089779292755358, 0.004116707488056058, 0.004463311658604419, 0.004863245054602056, 0.005019950105663084, 0.005111292599872651, 0.0050328244675445916, 0.004886511461492081, 0.005017059119637564, 0.004997550003214928, 0.004989853142609061, 0.004888243576205561, 0.004801721031771264, 0.005142349216675533, 0.0053550501391269115, 0.00510410900976245, 0.005113311675603742, 0.004865951202283446, 0.004739388247627576, 0.004314592960862043, 0.003932197365205607, 0.0036889365827877003, 0.003444247563217489, 0.0033695476656641706, 0.003779678994400599, 0.004182362477080399, 0.004650999598571368, 0.004964528816231351, 0.005246502668776329, 0.005150211093436487, 0.0051813375657147505, 0.005326590813316477, 0.00501407415865325, 0.004920848192186853, 0.005020741681762219, 0.005108871853087233, 0.004991922013198609, 0.005551866678436957, 0.005681472655730911, 0.005624204122058199, 0.005202581478369662, 0.00490495583623749, 0.0043628317352519584, 0.0037568042368143423, 0.0035018559432594323, 0.0035627004864066413, 0.003560172130774401, 0.003604382929642445, 0.003782708492731446, 0.003958167037361377, 0.004405696805281344, 0.004888234197579893, 0.004849378554876764, 0.005035728295111269, 0.005150565049279978, 0.005104177573029002, 0.005540331228404623, 0.005146813504207926, 0.004991504807148932, 0.0050371760815936415, 0.005174258383207836, 0.005598418288045426, 0.0056576481335463774, 0.00561832393839059, 0.005408391628077189, 0.0052292710408241285, 0.004705309149638305, 0.003924934489565002, 0.003854606161156092, 0.0038935040219155712, 0.003830335124052002, 0.003746046574771941, 0.003865490274877053, 0.004168222873979538, 0.0045871293840885514, 0.004915772256778214, 0.005072434696646597, 0.00492522147976003, 0.004978792784547765, 0.004963870334948144, 0.004955409293231536, 0.004709890770618299, 0.004888202349958703, 0.0051805005663287775, 0.005568883603736712, 0.005781789868618008, 0.006061759631832967, 0.005730308168750368, 0.0055273545529884146, 0.005050318950400666, 0.004505314632141857, 0.0041320733921015994, 0.0037557073980650723, 0.0034979193552635043, 0.0037461620721961097, 0.0036352203964434373, 0.003974040173135196, 0.004094756199243869, 0.004649079406159152, 0.004920019940715673, 0.005231951964023264, 0.005121117845618645, 0.005064423379922766, 0.00498326981229982, 0.004871188222923238, 0.004660839287914527, 0.0047034466283560495, 0.004866548640835444, 0.005578880008506938, 0.0059683185805929845, 0.006061498706153822, 0.005800490254423062, 0.0054633509277901724, 0.004921961696040911, 0.004376719066835311, 0.00393610914724284, 0.0037954515471031775, 0.003581690980473693, 0.003563708289302751, 0.0037463007418473766, 0.00403278474399164, 0.004356886520045223, 0.004787462849992179, 0.005179338649547787, 0.005143654461390953, 0.005203417442834235, 0.005153892139635152, 0.005114303176192244, 0.00504646961230832, 0.00478839952880454, 0.004711338394289699, 0.004911682972324793, 0.005442432018950797, 0.005865476365139558, 0.006157467255298909, 0.005776991413458904, 0.00537648513923766, 0.005215877640811999, 0.004586994881879395, 0.00404235177861292, 0.0038098588593210615, 0.003611933103919232, 0.003782482344031445, 0.003847756732676113, 0.004015496451997738, 0.004222327790973872, 0.004767228509347478, 0.005026217727591916, 0.005032992226639765, 0.0051856184936032845, 0.005070660243331873, 0.005025667638424633, 0.004771111450073196, 0.0049169687623427365, 0.0, 0.004725137860724068, 0.00480564403797717, 0.004993865191923319, 0.005382243541508231, 0.005436232552738047, 0.005416886729676188, 0.004777014387860352, 0.0048255785043644925, 0.004081842852408802, 0.004090331218562488, 0.00378104976817826, 0.003521792464859018, 0.0036283065618489215, 0.003818665737661915, 0.003988803567300145, 0.004483523199147563, 0.004696601941747573, 0.005206918843848881, 0.005231253931233336, 0.005154439277447777, 0.005107271378732522, 0.004862372011026066, 0.005097539245443387, 0.004771922511620435, 0.004800155668906229, 0.004886324331150043, 0.005186594367994167, 0.0055550364814704704, 0.00565254113064783, 0.00542892074907446, 0.005216026402108949, 0.0050842262523550985, 0.004506330112231061, 0.004262871158699087, 0.004073705404217544, 0.003562133424289835, 0.003499455612234611, 0.0037587992642927636, 0.004170545895025578, 0.004646029409170125, 0.004941082109950799, 0.005336110809450001, 0.005238846272634943, 0.0051019151317224926, 0.004828998520466023, 0.00470819320853546, 0.004974373055097931, 0.004975308413634935, 0.005266317039295838, 0.005489162450620279, 0.005606273008057806, 0.00603476714807901, 0.0061970275556501725, 0.0058349840239690235, 0.005192678736923442, 0.004639151581343363, 0.004229911816211891, 0.003727661961919841, 0.00375780482393585, 0.0033937487713780225, 0.003400171769633621, 0.003719857252709842, 0.0037474521895174925, 0.004410321140619574, 0.00505109832021614, 0.00506160098731807, 0.005046922423918226, 0.005300710721177051, 0.005104647840739084, 0.004974276083656935, 0.004902745159619985, 0.005039594632444682, 0.005189007878086687, 0.005840559146565768, 0.005924790523904985, 0.006041782063467494, 0.006054874048959406, 0.005728511142370623, 0.005014567400775691, 0.004479858189014036, 0.004064222403658478, 0.0038690888337760544, 0.0038101713160671666, 0.0038192317788082945, 0.003855643888760465, 0.004151893395194348, 0.004439198456142054, 0.004868610511159107, 0.005164087705238066, 0.0052260812748906515, 0.005049306708959293, 0.005295364532855441, 0.004976241631407976, 0.005325257379808529, 0.004981215539676753, 0.004904617253355752, 0.005133080934624669, 0.005474999665809228, 0.006018281474269119, 0.0059556619441451936, 0.00582564335486158, 0.0057773567703702745, 0.005185870554701607, 0.004927387470357575, 0.004290471577704514, 0.003894605250504856, 0.0036579206650162693, 0.0037227880322444513, 0.0037587839308041025, 0.004025131552347727, 0.0043915455477435165, 0.004973183367291931, 0.005602412946227073, 0.005438255876982902, 0.005057281453194344, 0.0055819722968782305, 0.0052582960278547575, 0.0060302188495155494, 0.003969113083640037, 0.004874700151948723, 0.0048366059153241445, 0.005174590517957408, 0.005237240077942745, 0.005935388138900985, 0.006375850801552381, 0.006218749794135666, 0.005833520305137985, 0.005325978611613252, 0.00473056992525788, 0.0039874605990664344, 0.0038460789847597175, 0.003587065463944717, 0.00384212944765237, 0.004264645875837948, 0.004969973892903938, 0.005856983835337711, 0.006181231788159266, 0.006313470979891048, 0.006097287985997557, 0.005694104539336737, 0.005355534257732001, 0.005274420505031954, 0.004712403572544698, 0.004584515000959549, 0.004766412751530095, 0.0048104263193712886, 0.005309031929686986, 0.006042498279882524, 0.006496377367343072, 0.005619222170751848, 0.005418471293122766, 0.005015661629991529, 0.005062499505228742, 0.004308572994534354, 0.0038880894398937347, 0.003538125785331658, 0.0034843298748529253, 0.003774099147478583, 0.003896742470805163, 0.004541861762097922, 0.004553179667775172, 0.004948038015149709, 0.0050269339456022605, 0.00522398911361471, 0.005050975431726277, 0.005007174429180125, 0.004833758244552214, 0.004670604547693902, 0.00477521510651887, 0.004939453753268834, 0.005239435739336397, 0.005820798534429634, 0.006094069145690364, 0.005673509972509797, 0.005375844111251002, 0.005187640280456208, 0.00476628984541101, 0.004247493846603608, 0.003794806926377327, 0.003435122854871529, 0.003587919312587277, 0.003811897320196127, 0.0042459415490763925, 0.00460744683733153, 0.004807733730818607, 0.005155657515164588, 0.005405463068510853, 0.005224147724524333, 0.005351078308428722, 0.005384714635929638, 0.005362056525935763, 0.0051377016971353075, 0.004941059319359612, 0.004966034655341646, 0.005026256144832193, 0.005442607412384369, 0.0059898202401797275, 0.005612531062072142, 0.005603529527930128, 0.0051493731726657554, 0.004544820351700367, 0.004496920773323335, 0.00424357787751253, 0.0036690501594786006, 0.003700340743778253, 0.0038846659058119253, 0.004159671170598417, 0.004794839922729552, 0.005004852590193807, 0.005163099925195087, 0.005645338914676821, 0.005432262412191398, 0.0050802949835114155, 0.005169574505964038, 0.0052347116826927985, 0.0052757424822272225, 0.0056125420409050475, 0.005578375783486106, 0.005944651628427074, 0.006010407699526147, 0.0061534279769882615, 0.005756457061668538, 0.005283251628717022, 0.004694029423550289, 0.0042271372620665245, 0.003995084263772031, 0.003916612465121526, 0.00385882298694225, 0.0039353658124175695, 0.00403048536977438, 0.0039523025458470164, 0.004692943486212761, 0.005099144811322234, 0.005182029052264465, 0.005496327599559573, 0.0053953892408097875, 0.005256712315751134, 0.004628655585945719, 0.005255300089578979, 0.004727544165215228, 0.005365188522646431, 0.006321448616075385, 0.005962859901186893, 0.0064913517773445605, 0.006403310018717368, 0.005985231247570929, 0.005536676822123271, 0.005652983876148263, 0.0053962798830303575, 0.0036360246130896887, 0.0034235996705107084, 0.004421584551524996, 0.003810299791511898, 0.0038131330853627154, 0.0038483466362599773, 0.005120205311426739, 0.0048344210780759, 0.005090949889906456, 0.005557094917028417, 0.005276073619631902, 0.0056143238257037814, 0.005700457782933553, 0.00584351804652435, 0.004893880732421001, 0.005475919992851946, 0.005248580353868141, 0.005350058838515571, 0.006083169087767963, 0.005703392826945841, 0.006319084795547654, 0.005231157508317081, 0.005381213703447174, 0.005027572682644346, 0.0042202572347266884, 0.004068212855323105, 0.003991170422748069, 0.0037477607718658665, 0.004077183917326014, 0.00408925876065761, 0.004650332253801002, 0.004960348232472058, 0.005144796809267916, 0.00597460791635549, 0.005407754333445995, 0.005265714189536858, 0.005391654498789258, 0.00495731680894397, 0.005033086804203971, 0.00511026991441738, 0.005391897414595909, 0.006005653123816428, 0.0066265552258310415]

i think that a good way for me to extract out the signal I am interested in would be to do a spectral analysis on the timeseries for my terms. The high frequencies should be the daily patterns which I want to get rid of and the lower frequencies should be what I’m interested in. I want to somehow ‘divide’ my observed signal for a term by the baseline daily signal.

This is my baseline’s original signal
baseline original
and this is my term’s original signal
term's original signal
and what i’m trying to do is get something like this in a general way without introducing artefacts. i.e remove the ups and downs that happen every day anyway and capture the general trend.
term tranformed

The naive way I thought of doing this is to first generate ffts for both using numpy(below).

baselines fft
baseline's fft

term1s fft
term1's fft

and then create a filter like below

fft2 = fft(term1, n=t)
mgft2=abs(fft2)  
plot(mgft2[0:t/2+1])
bp  = fft2[:]
for i in range(len(bp)):
if i>=22:
    bp[i] = 0
ibp = ifft(bp)

but from what i understand that introduces artefacts, changes the magnitudes and I am not sure how to pick a cutting point. I was hoping for some guidance with respect to implementation in numpy on a better way to divide out my baseline frequencies from my term’s frequencies.
thanks

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    It’s one thing to give the code, another to give an explanation. If you just want to look at the spectra, try the following:

    import numpy as np
import matplotlib.pyplot as plt

hh = np.hanning(len(term1)) # Use a Hann window to deal with spectral leakage
St = np.fft.rfft(hh*term1)
Sb = np.fft.rfft(hh*baseline)

FSample = 1.0/24            # Sampling frequency is 1/24th of a day
deltaF = 1.0/(len(term1)*FSample)  # Frequency resolution is 1/capT = 1/(NumSamples*FSample)

faxis = deltaF*np.arange(len(St))

plt.plot(faxis, np.log10(np.abs(np.array([St, Sb]).T)))

    Spectra of term1 (blue) and baseline (green)

    See Wikipedia’s explanation of spectral leakage and windowing for details on hh. The x-axis of the plot is calibrated in days. There are two thin peaks in the baseline spectrum at 1 and 2 day periodicity, and a broad peak around 1 day in the term1 spectrum.

    It wasn’t clear to me from your question if the baseline data already had the real data stripped out. If so, I think this shows that there isn’t much structure in one that’s distinguishable from the other.

    Another thing to keep in mind is that Fourier analysis assumes the signal is stationary. Depending on the physics of what you’re measuring this assumption may or may not be true. Certainly, the time-domain plot of term1 doesn’t look at all stationary, with a wild change in character starting around day 12:np.plot(np.arange(len(term1))/24.0, term1)

    Having said all that, if you have a way to characterize the baseline data, you might be able to apply noise cancelation algorithms like the Widrow-Hoff LMS algorithm. Wikipedia presents a very theoretical overview, I’m not sure where to find a more practical application oriented explanation.

    How did you come up with the term1/baseline separation in your example data?

    • 0