HE^j_ ~ddlZddlZddlmZmZddlmZmZmZmZddl m Z ddl m Z m Z ddlmZddlmZmZddlmZmZdd lmZmZmZd eed ed eefd Zdedededed df dZdedededed df dZdeded dfdZded e fdZ!d de"e#de$de$d e%e"e#e"e#ffdZ&de"e#d dfdZ'de"e#deded dfdZ(y)!N)datetime timedelta)OptionalListDictAny)settings)_log current_time)get_db_connection)send_push_notification_send_batch_summary)dispatch_discord_alertsdispatch_discord_edit)send_telegram_notificationsend_telegram_batchdispatch_telegram_edit first_seen removed_atreturnc|sy d}tjtj||}tjtj||}t||z }|dkryt |d\}}t |d\}}|dz} g} |r| j |d|r| j |d| s| s| j | dd j | S#ttf$rYywxYw) N%Y-%m-%d %H:%M:%SriQi<dhm ) timemktimestrptimeintdivmodappendjoin ValueError OverflowError) rrfmtt1t2deltadaysremhoursminutespartss ,/home/container/backend/scraper/processor.py_compute_listing_durationr1 s ! [[z37 8 [[z37 8BG  195%( cC& s)  LLD6 $  LLE7! % % LLG9A 'xx  &sA'C'-A9C''C98C9 product_code timestampprice save_percentc|jd|f|j}||d}||z }t|dkryd}|jd|||||fy)zWHelper to insert into price_history, calculating price_delta and preventing duplicates.z SELECT price FROM price_history WHERE product_code = ? ORDER BY timestamp DESC, id DESC LIMIT 1 Nr{Gz?z INSERT INTO price_history (product_code, timestamp, price, save_percent, price_delta) VALUES (?, ?, ?, ?, ?) )executefetchoneabs)cursorr2r3r4r5row last_price price_deltas r0_insert_price_historyr@$sw NN   // C V j( { t #   NN  5, DF event_typec0|jd||||fy)z2Helper to insert a stock event into stock_history.zp INSERT INTO stock_history (product_code, event_type, timestamp, price) VALUES (?, ?, ?, ?) Nr9)r<r2rBr3r4s r0_insert_stock_eventrE:s! NN  Iu 57rAc,|jd||fy)uInternal-only log of a raw 'product went missing' cycle, fired once per removal attempt regardless of whether it later matures into a confirmed removal or flap-recovers. Backs get_product_volatility's tiering — decoupled from stock_history's public 'removed' events, which now only get written once a removal is confirmed past the debounce window (see the "Matured removal notifications" block below).zZ INSERT INTO removal_attempts (product_code, timestamp) VALUES (?, ?) NrD)r<r2r3s r0_insert_removal_attemptrGAs NN  "$rActjttjz j d}|j d||f|jdS)uDCount raw removal attempts in the trailing window using timezone-safe local time cutoff. Reads removal_attempts (every attempt), not stock_history's public 'removed' events (confirmed-only) — a flappy product must still escalate tiers even though most of its flaps never mature into a public stock_history row.)r-rzo SELECT COUNT(*) FROM removal_attempts WHERE product_code = ? AND timestamp >= ? r)rnowrr FLAP_WINDOW_HOURSstrftimer9r:)r<r2 cutoff_tss r0get_product_volatilityrMMsZ )(2L2L"MMWWXklI NN  " $ ?? Q rAproducts is_first_run partial_scanct}g}g} |j}t}d}t} g} g} g} |D]3} | j d}|s| j || j d}| j dt j} t| j dd} t| j dd} t| j d d} t| j d dxsd } t| j d dxsd }| j d g}|r6tj|Dcic]}d|vsd|vs |d|dc}dnd }| j di}|j dxsi}|j dxsi}|j dg}d }t|tr#|j dr|j d}nyt|tr#|j dr|j d}nF|rDt|t r4t#|dkDr&|d}t|tr|j d}|r|j%drd|z}|j'd|f|j)}|dkDrt+d||z z dzdn|} |ut-d|d||j'd|||||||| ||||f t/||||| t1||d |||r|j3| | j3| |d!}!|d"}"|d#xsd}#|d$}$|d%}%|j'd&|||| |||||f |"dk(s|$d'k(rw|%?|j'd(|ft-d)|d|d*t5|d+t7|d+n6|$d'k7r6|j'd,|ft-d-t j8d.|d|n|#d/z}&|&t j8k\r|j'd0|f|j'd1|f|j)}'|'r|'drt;|'d|}(|(r|(| d2<t/||||| t1||d'||t-d3|&d4|d||s\|j3| | j3| n9|j'd5|&|ft-d6|&d7t j8d.|d|t=||!z d8kDst-d9|d:|!d;|d<|j'd=|| |ft/||||| |r|"d/k(s||!krd>nd?})| j3| |!||)d@6|sU|j'dA|j?}*|*D].}|dB}+|+| vst-dC|+d|dDtA||+},|,t jBk\rt jD}-n4|,t jFk\rt jH}-nt jJ}-|j'dE|||-|+f|d%d u}.|.stM||+|ttOjN|-z}/tOjPtOjN|-z}0tOjRdF|0}1t5|+dG|/Ht7|+dG|1I1|j'dJ|j?}2tOjTtOjV||}3|2D]}|dB}tYtA||d/z d},|,t jBk\rt jD}-dK}4n8|,t jFk\rt jH}-d}4nt jJ}-d}4tOjTtOjV|d%|}5|3|5z }6|6|-k\s|j'dL|ft1||dM|d%|d!t-dNt|6dO|d|dDdP|,dQ|4rdRn|,t jFk\rdSndTdU t5|dVt7|dV|r;|4r?t;|dW|}7|dB|dD|dX|d!|dYdZ|dB|d[|7d\}8| j3|8|j[dd]l.m/}9|9ja|jc|sqte| d t jfte| dMt jhtk| d tk| dM| rdd^l6m7}:| D] } |:| d  tq| ||fS#ttf$rd}YwxYw#ttf$rd}YwxYw#ttf$rd}YwxYw#ttf$rd }YwxYw#ttf$rd }YwxYwcc}w#|jcwxYw)_zJCompare freshly-scanned product list against database and process updates.r productCode productNameproductConditionwebPricerr8 finalPrice savePercent ratingStarN commentCountclassificationabF) ensure_asciimedia heroImage thumbnailgallery imageAddressz//zhttps:z SELECT current_price, active, stability_count, pending_state, pending_removal_since FROM products WHERE product_code = ? g?dzNEW PRODUCT DETECTED: z - a INSERT INTO products ( product_code, product_name, condition, first_seen, last_seen, removed_at, original_price, current_price, save_percent, active, specs, rating_star, comment_count, thumbnail_url, full_specs, stability_count, pending_state, pending_removal_since ) VALUES (?, ?, ?, ?, ?, NULL, ?, ?, ?, 1, ?, ?, ?, ?, NULL, 0, NULL, NULL) added current_priceactivestability_count pending_statepending_removal_sincea UPDATE products SET last_seen = ?, active = 1, removed_at = NULL, pending_removal_since = NULL, debounce_duration = NULL, product_name = ?, original_price = ?, save_percent = ?, specs = ?, rating_star = ?, comment_count = ?, thumbnail_url = ? WHERE product_code = ? back_in_stockz UPDATE products SET stability_count = 0, pending_state = NULL WHERE product_code = ? zFLAP RECOVERY: z) (reappeared before removal notification) restockedz UPDATE products SET pending_state = 'back_in_stock', stability_count = 1 WHERE product_code = ? zRESTOCK PENDING (1/z): z UPDATE products SET stability_count = 0, pending_state = NULL WHERE product_code = ? z SELECT timestamp FROM stock_history WHERE product_code = ? AND event_type = 'removed' ORDER BY timestamp DESC, id DESC LIMIT 1 _restock_durationz'PRODUCT BACK IN STOCK (confirmed after z cycles): z UPDATE products SET stability_count = ? WHERE product_code = ? zRESTOCK PENDING (/r7zPRICE CHANGE for z: z -> z INRzNUPDATE products SET current_price = ?, save_percent = ? WHERE product_code = ? price_drop price_hike)product old_price new_pricetypez SELECT product_code, product_name, condition, current_price, save_percent, first_seen, pending_removal_since FROM products WHERE active = 1 r2zPRODUCT REMOVED (SOLD OUT): product_namea UPDATE products SET active = 0, removed_at = ?, stability_count = 0, pending_state = NULL, pending_removal_since = COALESCE(pending_removal_since, ?), debounce_duration = COALESCE(debounce_duration, ?) WHERE product_code = ? z%I:%M %p cart_hold) expire_epoch) expire_timea SELECT product_code, product_name, condition, current_price, save_percent, first_seen, pending_removal_since, thumbnail_url FROM products WHERE pending_removal_since IS NOT NULL AND active = 0 Tz UPDATE products SET pending_removal_since = NULL, debounce_duration = NULL WHERE product_code = ? removedzREMOVAL CONFIRMED (z s elapsed): z (volatility: z, tier: z 2 (muted)10)sold_outr conditionr5z/p/ thumbnail_url)rRrSrTrVrWurlr_listing_duration) catalog_statedispatch_watchlist_alerts)9r r<r setgetaddr DEFAULT_CONDITIONfloatr% TypeErrorr!jsondumps isinstancedictlistlen startswithr9r:roundr r@rEr#rrSTABILITY_THRESHOLDr1r;fetchallrMTIER2_FLAP_THRESHOLDTIER2_DEBOUNCE_SECONDSTIER1_FLAP_THRESHOLDTIER1_DEBOUNCE_SECONDSTIER0_DEBOUNCE_SECONDSrGr localtimerKrr maxcommitbackend.scraper.catalog_stater init_from_dbclose_dispatch_notifications NTFY_TOPICNTFY_TOPIC_REMOVEDrbackend.notifier.discord_botr$_dispatch_price_change_notifications);rNrOrPconn new_listingsrkr<rIr' scanned_codes added_queue removed_queueprice_change_queuepcodenameroriginal_pricerfr5 rating_star comment_countrZs specs_jsonr^herothumbraritemr=calculated_save_percentdb_price was_activerhrirj new_countsh_row restock_durrB active_rows active_code volatilitydebounce_seconds is_first_holdrxexpire_time_structexpire_time_str pending_rowsnow_tssuppress_alerts pending_tselapsedduration dummy_productrrs; r0process_scanned_productsrZs  D!L "MBn!"%% "$ $& )+A55'D   d #55'D0(2L2LMI %!&quuZ';!< $ %aeeL!&< =  #$QUU=!%<=  ##AEE,$:;Ct  % #AEE.!$< = E UU#3R8N )7QA3!8q33Q"&*  EE'2&E99[)/RDIIk*0bEii 2.G M$%$((>*B $ 8 E4(UYY~-F % . 9 Z63w3Q-QUX,XZ[&\yE #{-dV3tf=> D)S#~}Vm +}mM N&fdCH_`#FD'3 N# ''*&&q)/ ] "%&7"8"=A #O 4 (+,C(D% 41H*VacprBFGH?m&F,8(#W&tfCv=fgh.dK@.t[A&/9(#W&283O3O2PPSTXSYY\]a\bcd%4a$7 $(D(DD"NN, #'* #NN, #' * &,__%6F%&).Gq SV.W #.=HA&9$:1&$]Tkl/osTab #J9+U_`d_eehimhn!op#/ - 4 4Q 7 + 2 21 5"NN, #,T!24!#4YKqA]A]@^^abfaggjkojp!qr}x/047,TF"XJd=/QUVWNNh&(?F*&$]Lcd'J!O6CX5M\S_ *11'()1)6$. 3gv NN  !//+K"!.1 m37 }CNH[G\]^"8 !LJ!X%B%BB+3+J+J(#x'D'DD+3+J+J(+3+J+J( NN$s$4kBD &))@%AT%IM$/ SI'*499;9I+I'J -1^^DIIKJZ':'*>':,/ ,<&)/&:'*>':!$S%8$9:),_)=-5 %M"((7i l  @""$  Wh6I6IJ y(:U:UV W5 y9  N )!W5!,-?@  &&s  * %!$ % * $ #  $ * #"  # * #"  # * % $  % Rn sBge 5e#e;-f f+,'g g g " g -F/gG/g;g gAg)BgFgBggA,g e ge  g#e84g7e88g;f gfgf($g'f((g+g<g?gggqueuecF|syddlm}ddlm}|D]}|d}|d}|d}|dd k(rBt |d | t |gd | t |d | ||d | |||| nt |d | t |gd | tjdy)uSends price-drop/hike alerts queued during the scan loop. Called after conn.commit()/conn.close() — see the queueing comment above — so the per-event network I/O and time.sleep(1) no longer hold the write transaction open.Nrr)notify_price_droprrrsrtrurp) ntfy_typers)rs)rsrtrqrm) rrbackend.notifier.pushrr rrrsleep)rrreventrrsrts r0rrs F7 ) +& +& =L ( "1 R #QC K &q,) L %a K a9 J "1 R #QC K 1 rArtopicc|syt|tjk\rt|||y|D]6}|j d}t |||t jd8y)Nr)rlisting_durationrm)rr NOTIFICATION_BATCH_THRESHOLDrrr rr)rrrrrs r0rrsV  5zX:::E9e4Auu01H "1 H U JJqMrA)F))rrrrtypingrrrrbackend.configr backend.logging_configr r backend.db.connectionr backend.notifier.ntfyr rrrrbackend.notifier.telegramrrrstrr1rr@rErGr!rMrrbooltuplerrrrAr0rsd (,,#53MWmm(3-SXVY]0FFFEFafFkoF,7c7s7s7[`7ei7 $# $# $$ $      T'tDzT'T'UYT'fklpqulvx|~ByCmCgDT'n T t2 4: # c d rA