HE^jM 2ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z m Z ddl mZddlmZddlmZddlmZddlmZddlmZdd lmZmZmZdd lmZdd lm Z m!Z!dd l"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,da-ej\Z/ej\Z0da1de2fdZ3de2fdZ4de5ddfdZ6de7de7de8de8ddf dZ9de7de7de8de8ddf dZ:de5ddfdZ;Gddejxejzj|Z?dZ@d"d ZAd"d!ZBy)#N)DictAny)settings)_log)get_db_connection)run_migrations)get_all_active_products)process_scanned_products) enrich_specsclean_ghost_listingsverify_and_enrich)backfill_psref_specs)send_startup_alertsend_once_complete_alert)LaptopTrackerHandler)send_telegram_notificationsend_telegram_batchsend_telegram_startup_alert!send_telegram_once_complete_alert)notify_new_listingnotify_back_in_stock notify_batchFreturnctjd}|sdddS tdt} |j }|j d|j ddk(}|jt\}}|dd dtjSt|}|dd dtjSt||| \}}t|||dt} |j }|j d |j d}|j d |j d} |jdt||| dtjS#|jwxYw#|jwxYw#t$r:} td| dt| dcYd} ~ tjSd} ~ wwxYw#tjwxYw)uRun a manual scrape process under scraper lock. Returns status dictionary. is_first_run is derived from the current row count (same rule run_scraper_loop uses at startup), not hardcoded — a manually-triggered scrape against an already-seeded DB must still queue and dispatch new/restock alerts like any other cycle, and only suppresses them on a genuinely empty database. Fblocking3A scrape or database update is already in progress.successerrorz [API] Manual scrape triggered...SELECT COUNT(*) FROM productsrNz,Failed to retrieve products from Lenovo API.z5Compare API unavailable. Cycle aborted (fail-closed). is_first_run partial_scan.SELECT COUNT(*) FROM products WHERE active = 1.SELECT COUNT(*) FROM products WHERE active = 0T)r scanned_count active_count removed_countz[API] Manual scrape failed: ) scraper_lockacquirerrcursorexecutefetchonecloser releaser r "_dispatch_notifications_backgroundlen Exceptionstr) acquiredconnr,r#productsr$ new_listings back_in_stockr(r)es /home/container/backend/main.pytrigger_manual_scraper<(s##U#3H  +`aa* /0 " [[]F NN: ;!??,Q/14L JJL!8!:,  $/]^: 7%X.  $/fg2 1'?xVbq}&~# m*< V[\!" [[]F NNK L!??,Q/L NNK L"OO-a0M JJL ](*   C JJL* JJL 3 +A3 /0 3q622 3 skF*7F,#F*$F* )F*4AF F*FF*F''F** G-3G(G-G0(G--G00Hcltjd}|sdddS tdt}dt ||dtj S#t $r:}td|dt|dcYd }~tj Sd }~wwxYw#tj wxYw) zIRun a manual ghost cleanup under scraper lock. Returns status dictionary.Frrrz [API] Ghost cleanup triggered...T)r cleaned_count cleaned_codesz[API] Ghost cleanup failed: N)r*r+rr r2r0r3r4)r5r?r:s r;trigger_ghost_cleanupr@`s##U#3H  +`aa  /0,.  /*   3 +A3 /0 3q622 3 s/#A B B;B<BBBB3signumcdatd|dtr4tjtj dj yy)z,Handle SIGTERM/SIGINT for graceful shutdown.TzReceived shutdown signal u. Shutting down gracefully…targetdaemonN)_shutdown_requestedrhttpd threadingThreadshutdownstart)rAframes r;_handle_shutdownrMssB $VH,I JK t<BBD r8r9r#silentc|s|ry|rQt|tjk\r t|dn(|D]#}t |dt j d%|rQt|tjk\r t|dn(|D]#}t |dt j d%|r-t|dk\r t|dn|D] }t||r.t|dk\r t|dy|D] }t|yy)zIDispatches Telegram notifications and Web Push notifications for updates.Nrestockadded new_listingr9) r2rNOTIFICATION_BATCH_THRESHOLDrrtimesleeprrr)r8r9r#rOps r;dispatch_all_notificationsrZ{sv } !F!F F  y 9"*1i8 1 #  |  E E E  g 6!*1g6 1 "  |  ! } 5!"1%" }  "  8"$Q'# rNc`tjt||||fdjy)uQRuns dispatch_all_notifications() (Telegram + Web Push) off the scrape loop's critical path. Both do blocking network calls per recipient — Telegram even sleeps 1s between individual sends below NOTIFICATION_BATCH_THRESHOLD — so inline they can add real seconds to every cycle. Discord doesn't need this: dispatch_discord_alerts/_edit already hand off via asyncio.run_coroutine_threadsafe onto the bot's own event loop thread, non-blocking already. Unlike the PSREF backfill below, this is never guarded/skipped even if a previous cycle's dispatch is still running — a specific restock/price-drop notification is a one-time event tied to this cycle's diff; skipping it isn't "retry later," it's silently dropping a user-facing alert. See DECISION_LOG.md's 2026-07-18 entries for the reasoning. TrDargsrEN)rHrIrZrK)r8r9r#rOs r;r1r1s-")M< @ egrNcyclectjdstddyfd}tj|dj y) uJRuns the PSREF backfill off the scrape loop's critical path — a slow or fully-down PSREF must never delay real Lenovo polling/notifications (see DECISION_LOG.md's 2026-07-18 entries). Skips spawning a new run if a previous one is still in flight — only possible if an unusually large batch of new machine types took longer than one POLL_INTERVAL to process. Safe to skip: unlike a notification, an MT left uncached this cycle is picked up automatically by the next successful run's own "not yet in psref_specs" query, so nothing is permanently lost. Fr[Cycle zG] PSREF backfill still running from a previous cycle, skipping trigger.Nc  t}|ds|drtdd|dd|ddtjy#tjwxYw)Nenrichedfailedr`z] PSREF backfill: z enriched, z failed.)rrpsref_backfill_lockr0)resultr^s r;_runz,_run_psref_backfill_background.._runsl *)+Fj!VH%5eW$6j)*+fX6F5GxQ ' ' )  ' ' )s /AATrC)rdr+rrHrIrK)r^rfs` r;_run_psref_backfill_backgroundrgsK  & & & 6 wugd ef *D.446rNceZdZdZy)ThreadingHTTPServerTN)__name__ __module__ __qualname__daemon_threadsrNr;ririsNrNric t} |j}|jd|jd}|j |dk(}d}t d|r t dn1t d|d|j st|t||j spddl m }m }|rat dtd D]=}|} | r| jr t d n"tjd ?t d t s|d z } t"5t%\} } | Rt'| } | t d|dnTt)| |xs |j | \} } t+| | |xs |j |j t-|t} |j}|jd|jd}|jd|jd}|j t d|d|d|dt/| |rot dt/| d|j s(tt/| tt/| d}n t d|dt0j2ddddtt0j2D]}t rntjd !t st dy#|j wxYw#|j wxYw#1swYxYw#t4$r}t d|d|Yd}~d}~wwxYw)z9Scraper daemon loop that runs inside a background thread.r!ru'Initializing background scraper loop…u8Database is empty. Performing initial background scan…zDatabase contains z records. Ready to monitor.is_discord_enabledget_discord_botzHWaiting for Discord bot to connect and be ready before starting scans...<z-Discord bot is ready. Starting scraper scans.g?zWWarning: Discord bot is still not ready after 30 seconds. Proceeding with scans anyway.rRNr`z8] Compare API unavailable. Skipping cycle (fail-closed).r"r%r&z] Scan complete. Active: | Removed: | Scanned: ,Initial scan complete. Database seeded with products.Fz+] Failed to retrieve products. Retrying in us…z] Unexpected error: z Scraper background loop stopped.)rr,r-r.r/rrOrrbackend.notifier.discord_botrqrrrangeis_readyrWrXrFr*r r r r1rgr2r POLL_INTERVALr3)r]r6r,db_countr#r^rqrr_botr7r$r8r9r(r)r:s r;run_scraper_looprs  D67??$Q' q=L E 23 GH !(+F GH{{ x ( ' 1 ;;T   [ \2Y%'3<<>HI 3 no!  4 ;)@)B&,'0:H'wug-efg6N$)5)D)573 m;(-9TVZVaVa7u=02)%)[[]F"NN+[\+1??+.start_botSsF 8))H667 7 7 8 21#6 7 7 8s6A"/-/A/ AA A AAz[Discord] Event loop error: return_exceptionsT) rxrqrrrasyncionew_event_loopset_event_looprun_until_completer3 all_taskscancelgatherr/)rqrrrloopr:pendingtaskr~s @r;run_discord_botrEsW   PQ AB  C 8  ! ! #D 4    , ''-G  ''(XSW(XY JJL 1 +A3 /001    ''-G  ''(XSW(XY JJL   sb!C8A"C? C<$C72D7C<<D? D  D FA"E32F3 E?<F>E??Fc $tjtjttjtjtt j d}|j ddd|j ddd|j d dd |j}td d l m }|jd d l m }m}||j}t!} |j#}|j%d|j'd }|j)|d k(} |j*rt-d t/\} } | t-dt1j2dt5| } | t-dt1j2dt7| | xs |j8| \} } t;| | | xs |j8|j8t!} |j#}|j%d|j'd }|j%d|j'd }|j)t-d|d|dt=| | rMt-dt=| d|j8sMt?t=| tAt=| n$|j8stC||tE||t-dytIjJtLjNdd}|r+tQjRtTd }|jWtQjRtX|fd!}|jWd"tLjZf}t]|t^a0t-d#tLjZ t`jcda3t`jit-d%|jkd&'|r|rtm|d(rtm|jnd)rj|jnjqrPt-d*d dl9}tujv|j)|jn} |jyd&'t-d/t1j2d y#|j)wxYw#|j)wxYw#tF$r.}t-d|t1j2dYd}~*d}~wwxYw#td$rt-d$YdwxYw#ttjz|j|jzf$rt-d+Yttj~|j|j~t~f$rt-d,YtF$r/}t-d-t|jd.|Yd}~Cd}~wwxYw#da3t`jit-d%|jkd&'|r;|r8tm|d(r+tm|jnd)r|jnjqrt-d*d dl9}tujv|j)|jn} |jyd&'n#ttjz|j|jzf$rt-d+Ynsttj~|j|j~t~f$rt-d,Yn9tF$r.}t-d-t|jd.|Yd}~nd}~wwxYwt-d/t1j2d wxYw)0NzTrackFurb India Laptop Tracker) descriptionz--once store_truez=Run a single sync scan and exit instead of running in a loop.)actionhelpz--silentzXRun silently without sending any push notifications (use with --once or on initial run).z--sync-commandsz%Sync Discord slash commands globally.r) catalog_state)init_discord_botrq) sync_commandsr!uRunning single scan (--once)…z0Failed to retrieve products. Exiting with error.rRz "HlGH (2HST *B)8T[[)+ 'L- '|]LD_TXT_T_aealal m$&D OP%03 OP & 1! 4  *<. ]OS_`cdl`m_no pCCM?R\]^{{&s8}5/H >{{,\=I5lMR -.KK##d3N"))N%%-=TGTXYN(--(N 0D EE ?  OP  #  HIA&  SWS&-A,/CHH4G4G4I 4 5 %22399;IC L 1 % Z  A D   8< = HHQKK 6 ? =>?**J,>,>,M,MN 756((**<*<*I*I<X MKL L2473C3C2DBqcJKK L#  HIA&  SWS&-A,/CHH4G4G4I 4 5 %22399;IC L 1 %**J,>,>,M,MN 756((**<*<*I*I<X MKL L2473C3C2DBqcJKK L Z  s94S B+S7AS"B S7T1U S"S44S77 T.#T))T.1U W7U  W7 4W48W4=W4$W//W47B:^2[^4]+9^;8]+3^5]+=$]&!^&]++$^)rN)CrrrWrrrH socketserver http.serverhttprtypingrrbackend.configrbackend.logging_configrbackend.db.connectionrbackend.db.migrationsrbackend.scraper.apir backend.scraper.processorr backend.scraper.enrichmentr r r backend.scraper.psrefrbackend.notifier.ntfyrrbackend.server.handlerrbackend.notifier.telegramrrrrbackend.notifier.pushrrrrFLockr*rdrGdictr<r@intrMlistboolrZr1rgThreadingMixInserver HTTPServerrirrrrnrNr;rsO #'307>\\6N7 y~~ $inn& 6t6pt&ESEDE$(T$($$(VZ$(dh$(mq$(N'+;?IM 07#7$7@,55t{{7M7Mf-P%VGrN