HE^jPUddlZddlZddlZddlZddlmZmZddlm Z ddl m Z  ddl m Z dZdaee j"ed<d e j"fd Zd e j"fd Zd ed ed e j,fdZddeded eefdZd eeeeeffdZy#e$r ddl Z dZYuwxYw)N)OptionalList)settings)_log)requestsTF_sessionreturnct6trtjdatStjatS)z   s ( 44methodurlc t}tr|jdd |j||fi|S#t$rL}t d|d t }|j||fi|cYd}~S#t$r}||d}~wwxYwd}~wwxYw)zPMake an HTTP request with automatic session recovery on failure (Arch 4 helper).r r zHTTP connection failed: z). Recreating session and retrying once...N)rr setdefaultrequestrrr)rrkwargssessioneretry_es r_requestr (snG-5!wvs5f55 ! 's*S TU !')G"7??639&9 9 !q  ! !s98 B BA4.B 4 B=BBBB page page_sizec|tj}tjt|t|dd}t j |}t jjt jj|}dtjd|}tj|z}d}tdtjzD]Y} td|tjd } | jd k(r_| j } | jd d k(r| jd icSd | jd d| jd}nd| j}|tjkstj"r0|t%tj"krtj"|} n&tj"rtj"d} nd} t'd|d|d| d|dzdtjd t)j*| \t'd|ddtjzd|y#t $r} d| }Yd} ~ d} ~ wwxYw)z;Fetch a single page of products from the Lenovo Outlet API.Nv2) pageFilterIdr!pageSizeversionz?pageFilterId=z&loyalty=false¶ms=GET)headerstimeoutstatusdataz API error z: msgzHTTP zNetwork error: g@Page z fetch failed (z). Retrying in us… (attempt /)z fetch failed after z attempts: )r PAGE_SIZEPAGE_FILTER_IDstrjsondumpsurllibparsequoteAPI_URLrangeMAX_PAGE_RETRIESr HEADERS status_codegetr RETRY_DELAYSlenrtimesleep) r!r" params_dict params_jsonparams_encoded query_stringfull_url last_errorattemptresponser/rdelays rfetch_productsrP7sO&& !//D  N K **[)K\\'' (:(:;(GHN 001 !" $ ,.H $JX6667 /x9I9ISUVH##s*}}88H%,88FB//#-dhhx.@-ADHHUOCT!UJ$X%9%9$:;  X.. .$$3x7L7L3M)M --g6&& --b1 5oj\~^ehi^i]jjkltmFmFlGGHI J JJu /82 5*1x/H/H+H*IU_T` ab  /*1#.J /sA!I;6I I-I((I-c d}d}ttdd}tdtj}|s t dy|j d g}|r dd l}|jj|jjtj}|jj|d }t|d d 5}tj||ddd d d t d||j dg} | rt!| t"s t dy| r| dj dnd } | s t dyt%gdt"t&dd ffd } | | |j dd} | r&t)j*| tjz nd}|dkDrt d| d|dt-d|dzD]}t/j0tj2t|tj}|s3|dz }d}t d|d|d|d ||kDrt d!|d"yv|j dg}|r|dj ds3|dz }d}t d|d#|d|d ||kDrt d!|d"y| |dj dg|fS#1swYxYw#t$r} t d| Yd } ~ d } ~ wwxYw)$zlRetrieve every active product across all paginated API pages. Returns (all_products, partial_scan). FrMAX_FAILED_PAGESr()r!r"z4First page fetch failed. Aborting scrape completely.)NT facetGroupsNzfacet_groups.jsonwzutf-8)encoding)indent ensure_asciiz!Saved live filter facetGroups to zError saving facetGroups: r/z._add_uniques?A55'DJ.t$##A& rcountzScaling fetch: Found z items across z pages.Tr2z failed (r3z).zFailed pages exceeded limit (z). Aborting cycle.z returned empty products ()getattrrrPr5rrBospathdirnameabspathDB_FILEjoinopenr8dumpr isinstancelistsetdictmathceilr>rErFINTER_PAGE_DELAY) partial_scanfailed_pages_count max_failedr/ facet_groupsredb_dirfacet_groups_pathfg_fileritemsfirst_page_productsrbrc total_pagesr^ page_data page_itemsr`ras @@rget_all_active_productsrhsL#5q9J qH,>,> ?D  CD88M2.L 3 WW__RWW__X5E5E%FGF " V5H I 'w?7 ,N@ 45F4GH I HHVR E  5$/ KL6;%(,,z2  JK5J!L'd4j'T'#$ HHWa E;@$))EH$6$667aKQ $UG>+g NOq+/*A JJx00 1&A9K9KLI"a'"# uQCy);(rs  !#'"M (,(8## $+h&& 8++  !S !s !1B1B !////bM&xT ';T'A!BM&wMsB B%$B%