HE^jy~ UdZddlZddlZddlZddlZddlZddlmZm Z m Z m Z m Z ddl mZddlmZddlmZmZddlmZdZe eefZd e ed efd Zd e eefd efd Zde ed e eeffdZded e e eeffdZdeded e e eeffdZdkde e ed e eeffdZ e eefZ!e eefZ"gdZ#gdZ$e e eefe%d<gdZ&e e eefe%d<hdZ'ddddddd d!dd"dd#d#d#d$d%Z(d&ed'ed(ed)e ee e!fd*e ee e!fd df d+Z)ejTd,ejVd-fejTd.ejVd/fejTd0ejVd1fejTd2ejVd3fejTd4ejVd5fejTd6ejVd7fejTd8ejVd9fejTd:ejVd;fejTdejVd?fejTd@ejVdAfg Z,e e ejZefe%dB<ejTdCejVdDfejTdEejVdFfejTdGejVdHfejTdIejVdJfejTdKejVdLfejTdMejVdNfejTdOejVdPfejTdQejVdRfgZ.e e ejZefe%dS<dTedUed efdVZ/d&ed'ed(ed)e ee e!fd df dWZ0dXe eefde eefd)e ee e!fd*e ee e!fd df dYZ1de eefd)e ee e!fd dfdZZ2dXe eefde eefd)e ee e!fd dfd[Z3d ed efd\Z4hd]Z5de eefd^e ed e e eeffd_Z6d`e ed^e ed e efdaZ7d e eefd e e!fdbZ8d ed e e9fdcZ:d ed e efddZ;de eefdXe eefd e e eeffdeZdXe eefdie e eefd e e e"e e"e effdjZ?y)luyPSREF (psref.lenovo.com) spec-sheet enrichment. Fills gaps our own outlet scrape never reports (chassis/security/ports/certs) by resolving each catalog product's Machine Type (first 4 chars of `product_code`) against Lenovo's undocumented PSREF JSON API and caching the flattened spec sheet, keyed by MT (a spec sheet covers a whole product line, not one SKU). Field-by-field research (which PSREF fields are flat-safe, which are genuinely ambiguous, the two deterministic joins) lives in the `project_psref_feasibility` memory — this module builds against that directly. Response shape below was confirmed against the real API this session (not guessed from the memory's prose description), e.g.: Suggest: {"data": [{"ProductKey": "...", "MachineType": "21JR", "info": {"page": "https://psref.lenovo.com/l/..."}}]} LoadSpecData: {"data": {"GeneralSpecData": [{"GeneralSpecJson": ""}]}} GeneralSpecJson parses to {"data": {"SpecData": [ {"L1": "Performance", "L2": [{"L2": "Chipset", "Features": [ {"FName": "Chipset", "Optional": "0", "FVs": [ {"FVGs": [{"FVGItem": [{"Att": "DefaultDescription", "AttV": "AMD SoC..."}]}]} ]} ]}]} ]}} ToS: Lenovo's Terms of Use bans "high volume" automated harvesting. This is 2 calls per unique MT, once, cached forever (~120 calls total as of this writing) — flagged to the user before this runs against the live site for real, same as the memory's own caveat. N)AnyDictListOptionalTuple)settings) db_connection)_log current_time)_requestg?sreturnc|sytj|}tjdd|tjtj z}tjdd|}tjdd|j }|S)u^Some PSREF description fields (e.g. Docking) embed a live tag pointing at their own accessories page — converted to `[[text|url]]` (not markdown's `[text](url)`: some of these real URLs contain literal parentheses, e.g. the BIOS Simulator link's "...(21Q6,21Q7)" segment, which breaks naive markdown-link parsing) so the frontend (lib/specValue.tsx) can render a real clickable link, instead of dead "please visit here" text with the href silently discarded. Any other/nested tags (a `` almost always wraps the anchor's own text) are stripped as plain formatting noise.z']*href="([^"]*)"[^>]*>(.*?)z [[\2|\1]])flagsz<[^>]+>\s+ )htmlunescaperesubISstripr s (/home/container/backend/scraper/psref.py _clean_textr6sr  aA 9<RTRVRVY[Y]Y]R]^A z2q!A vsA$$&A Hattrscd|vr|dS|jDcgc] \}}|dk7s |}}}dj|Scc}}w)uRender one FV group's attrs as display text. Single-attribute features use the generic "DefaultDescription" key; composite ones (Weight, Dimensions) carry a "Models" qualifier alongside the real value — skip it here, it's metadata, not part of the visible value.DefaultDescriptionModelsz, )itemsjoin)rkvpartss r_fv_textr(HsP u$)** ;;= :=41aAMQ=E : 99U  ;s AA spec_datac `i}|D]%}|jdd}|jdgD]}|jdd}|jdgD]}|jd}|sg}|jdgD]|} | jdgD]e} i} | jdgD]8} | jd } t| jd }| s1|s4|| | <:| sU|j| g~|s||t||jd d k(|d ||<(|S)u{Flatten PSREF's L1 -> L2 -> Feature -> FVs -> FVGs -> FVGItem nesting into {FName: {l1, l2, num_fvs, optional, values: [attrs_dict, ...]}}. Keyed by the Feature name (FName) directly, not nested under category — resolve_specs_for_laptop looks fields up by name; which display category each ends up in is decided there, independent of PSREF's own L1/L2 taxonomy.L1rL2FeaturesFNameFVsFVGsFVGItemAttAttVr1)l1l2num_fvsoptionalvalues)getrappendlen)r)flatr5l1_namer6l2_namefeaturefnamer9fvfvgritemattattvs r_flatten_spec_datarGWs4 "$D&&r"&&r"BffT2&G66*b1 G,/1!++eR0B!vvfb102$'GGIr$:D"&((5/C#.txx/?#@D"t-1c %; !"MM%0 21!!"6{ ' J 73 >$ U #2#8 KrmtcPtjdtjj d} t d|tj d}|jdk7rtd|jdy |jjd xsg}tfd |Dd }|xs |r|d nd }|r|jd sy |d |jdxsijddS#t$r}tdd|Yd }~y d }~wwxYw)uResolve a Machine Type to a PSREF `ProductKey` + spec-sheet page URL via the `Suggest` search endpoint. 100% hit rate across all 119 MTs ever seen in the catalog (per the feasibility sweep) — a miss here means a genuinely new MT PSREF hasn't indexed yet, not a bug.z1/api/search/DefinitionFilterAndSearch/Suggest?kw=z0&limit=6&IsPreviewProduct=true&SearchType=NormalGETheaderstimeoutz[PSREF] Suggest HTTP  for MT Ndatac3LK|]}|jdk(s|yw) MachineTypeN)r:).0rrHs r z'resolve_machine_type..s"GAAEE-,@B,Fas$$r ProductKeyinfopage) product_keypage_urlz[PSREF] Suggest error for MT : ) rPSREF_BASE_URLurllibparsequoter PSREF_HEADERS status_coder jsonr:next Exception)rHurlresponseresultsexactrDes` rresolve_machine_typerk}s,  " " #$||!!"%&&V XE30F0FPRS   3 & ()=)=(>hrdK L--/%%f-3GGN9wD488L1 -&)/R44V<    ,RD1# 67s%AD>AD)D D% D  D%rZctjdtjj |} t d|tj d}|jdk7rtd|jd|y|jjd xsijd xsg}|std |d |d y|djd}|sytj|jd xsijdxsg}t|}|std|d |d y|S#t$r}td|d|Yd}~yd}~wwxYw)zFetch and flatten the canonical per-product-family spec sheet (the same data that powers PSREF's own Spec tab and Compare feature).z-/api/product/Compare/LoadSpecData?ProductKey=rJrKrLrOz[PSREF] LoadSpecData HTTP rPNrQGeneralSpecDataz0[PSREF] LoadSpecData: no GeneralSpecData for MT  ()rGeneralSpecJsonSpecDataz,[PSREF] LoadSpecData: empty SpecData for MT z"[PSREF] LoadSpecData error for MT r\)rr]r^r_r`r rarbr rcr:loadsrGre) rZrHrfrggeneralraw_json spec_listr=rjs rfetch_load_spec_datarvse $ $ %%RSYS_S_SeSefqSrRs tCE30F0FPRS   3 & -h.B.B-C8B4P QMMO''/52::;LMTRT CB4r+VWX Y1:>>"34jj*..v6<"AA*MTRT !), ?t2k]RST U  1"Rs ;<s2AD;1D EFX'91= w'EAr "L1, vh&KJ<X==FKH@@  >/3 ./aKF8MJK0-1H"M  ./$aKF  ]+Z(JJyu= N   A 7t2c)n=MYWXI(R 0 +fX NO$7} F^ >! 22$b<== >s snAH" G0+G02H"A)H"9:G53H"4/G5#H"$A*G5H"0H"5 H>HH"HH""H+) ProcessorMemoryStorageGraphicsDisplay ConnectivityPhysicalPlatformPortsSecurityDurability & Certifications))Chipsetr)z Audio Chipr)Speakersr) Microphoner) Card Readerr)Dockingr)z Max Memoryr)zMax Storage Supportr)z Storage Slotr)z Security Chipr)zPhysical Locksr) ThinkShieldr)zScreen-to-Body Ratior)zMonitor Supportr)z Case Materialr)zDimensions (WxDxH)r)z Mil-Spec Testr _FLAT_FIELDS))zStandard Portsr)zOptional Portsr)z BIOS Securityr)zOther Securityr)Green Certificationsr)Other Certificationsr _ENUM_FIELDS>SIM Card Case Color Memory Type Battery Life Storage Type Base Warranty Power AdapterOperating SystemProcessor FamilyWLAN + BluetoothSystem ManagementFingerprint ReaderKeyboard BacklightBundled AccessoriesWWANCameraWeightBatteryrrKeyboardr Touchscreen Memory SlotsSmart Card Readerrrrrrrrrr)rrzRemovable Storagez Multi-Mediaz Input Device MechanicalNetworkrrrzOperating EnvironmentSustainabilityrrrfieldcategorylabelextra referencecZd|dDDcgc]}|s| }}|syt|dk(r(|j|gj||ddyttj |}|j|gj|dj |dycc}w)uUniversal rule for every singular field: check the live num_fvs for this specific product. ==1 -> confirmed, flat row. >1 -> all documented variants, joined (never a bare em dash — see feedback_no_em_dashes memory).c32K|]}t|ywNr(rTr&s rrVz!_emit_singular..y>o!or9Nr}rrvaluez or )r< setdefaultr;rrrr$)rrrrrttextsuniqs r_emit_singularros?eHo> D>1!Q>E D  5zQ 2&--a.QRDMM%()Xr*11EFKKX\L]2^_ Es B(B( thunderboltzThunderbolt / USB-Cz usb-?c|usb4zUSB-Czusb-?a\bzUSB-AhdmiHDMIzdisplayport|\bdp\b DisplayPortzethernet|rj-?45Ethernetz*headphone|microphone|audio jack|combo jackz Audio Jackz smart cardrzcard reader|sd cardrznano-?sim|sim cardzSIM Slotz\bvga\bVGA_PORT_LABEL_PATTERNSz energy starz ENERGY STARepeatEPEATz\berp\bErProhsRoHSz\btco\bz TCO CertifiedeyesafeEyesafeut[üu]vuTÜV Rheinlandzmil-stdzMIL-STD_CERT_LABEL_PATTERNS field_nametextcd|jvrtnt}|D]\}}|j|s|cS|S)Nport)lowerrrsearch)rrpatternspatternrs r_enum_item_labelrsB'-1A1A1C'C#I]H" >>$ L# rcd|dDDcgc]}|s| }}|D]1}|j|gjt||||d3ycc}w)uwOne row per item (Product Details/Compare's "separated" convention), each tagged with the source PSREF field name via "field" — Product Details ignores it, but Compare's fixed-row-per-laptop table can't show a variable-length per-item list per column, so it groups back by "field" into one compact joined row instead (see lib/compare.ts's psrefFieldJoined).c32K|]}t|ywrrrs rrVz_emit_enum..rrr9)rrrN)rr;r)rrrrrrrs r _emit_enumrsa ?eHo> D>1!Q>E D 2&--&ud3dU S  Es AAlaptopc&|jdxsd}|jd}tjd|tj}|r|rdj d|dD}tjd|tj}|r9|j d d |j d d|j d d } n|j } |jd gjd| dy|rt|d d||yy)aAlways emits a row. Never guesses stick count/size. If outlet's Memory string already states a soldered amount, pair it with PSREF's slot-count + DDR generation (not outlet's own socketed-capacity figure, which would falsely imply the exact populated stick size).memoryrrz(\d+\s*GB)\s*Solderedrc32K|]}t|ywrrrs rrVz%_emit_memory_slots..sC?ahqk?rr9z6\b(one|two|three|four|\d+)\s+(DDR\d)\s+SO-?DIMM\s+slotr}z Soldered + z SO-DIMM slotrrN) r:rrrr$grouprrr;r) rr)rrmemrsoldered psref_textslotrs r_emit_memory_slotsrs **X  $"C MM. )Eyy13=HEXXC5?CC yyRT^`b`d`de ~~a()djjm_AdjjQRm_TabEIIKE 2&--QV.WX uhyI rc|jd}|sy|ddk(rCt|ddjj}|j drdnd }nd}|j d gj d|d y) uDecisive Yes/No, never a hedge — a buyer can't retrofit one either way, so an occasionally-wrong "No" beats a technically-safer "may vary".rNr7r}r9rnoNoYesrr)r:r(rr startswithrr;)r)rrrrs r_emit_smart_card_readerrs MM- .E  Y1h*+11399;-5 Z$++6ITY,Z[rcN|jdxsi}|jdrd}nX|jd}|r)djd|dDjnd}tjd |rd nd }|j d gj d|d y)uThree real states. Outlet's `SIM` key present is itself a fully confirmed signal (same reliability as Fingerprint Reader's presence/absence) — its absence means "Built-in" is ruled out, not "unknown". full_specsSIMzBuilt-inrrc32K|]}t|ywrrrs rrVz_emit_wwan..rrr9rzslot|antenna|upgradablezHas a WWAN slotrrrN)r:r$rrrrr;)rr)rrrrrs r _emit_wwanrsL)/RJ~~e f%JO>eHo>>DDFUW%'YY/I5%Q!W[ ^R(//&50QRrctj|xsd}tjdd|}tjdd|}tjdd|j j }|S)Nru [®™©]z \([^)]*\)rr)rrrrrrrs r_normalize_cpursc ag2A |R#A |R#A vsA$$&,,.A Hr> Max FrequencyBase FrequencyCacheCoresThreadsProcessor NameProcessor Graphicsoutlet_processorc|jd}|r|syt|}|dD]'}|jd}|st||vs%|cSy)uMatch outlet's known Processor string against PSREF's per-SKU Processor Name to unlock Cache/Cores-Threads/NPU/iGPU-TOPS — none of which outlet reports. Deterministic: the CPU SKU token is exact, not fuzzy.rNr9r )r:r)r)rr outlet_normrnames r_match_processorrs\ MM+ &E ( !12Kxyy)* N4(K7L! rpsref_spec_data_jsonc|r|sy tj|}t ||}|sy|j d}|rd|SdS#ttf$rYywxYw)uResolve a laptop's integrated GPU name from PSREF's per-SKU Processor Graphics attr (the same join _match_processor already does for Cache/ Cores-Threads/NPU-TOPS), so it works for any CPU SKU PSREF has indexed — not a fixed table. Confirmed against a live call for MT 21JK (ThinkPad E14 Gen 5 Intel): Core i3-1315U -> 'Intel® UHD Graphics', Core i5-1335U/ i7-1355U/i7-1365U -> 'Intel® Iris® Xe Graphics' — exact match to what outlet itself reports for every sibling SKU in the same catalog, so this is read as ground truth, not guessed. See the module-level comment above for why this is safe to call for *any* CPU tier despite that not being true of the single-field dGPU-vs-iGPU question.Nrz Integrated )rcrr TypeError ValueErrorrr:)rrr)rgfxs rinfer_igpu_from_psrefr'sr '7*.**5I*J  Y(8 9E  ))( )C"%[ /4/ z "sAAAcg}|jdr|jd|dd|jd|jd}}|r|r|jd|d|d|jD]H\}}|tvrt j d|tj s5|j||dJ|S)Nr rr r zCores / Threadsz / znpu|tops)r:r;r#_PROCESSOR_JOIN_SKIP_ATTRSrrr)rrowscoresthreadskeyvals r_processor_join_rowsr!?sD yy gg?@YYw'9)=7E  /UG3wi:PQRKKMS , ,  99[#rtt , KK#4 5 " Krcrtjd|xsd}|rt|jdSdS)Nz(\d+(?:\.\d+)?)rr})rrfloatrr ms r _parse_inchesr&Ns1 $ag2.A !5 +t+rctjd|xsdtj}|r4tjdd|j dj SdS)Nz(\d{3,4}\s*x\s*\d{3,4})rrr})rrrrrrr$s r_extract_resolutionr(SsH ,ag2rtt23L::l+1r88:@@BJdKxUYYvr23!%))L""=> <3;#d[.@*AD*HCS]L] ii(.B557 Z B[ %B L ii'-2446<<> :* *C  ! rcg}|jdr|jd|dd|jd}|r|jd|d|jd}|r|jd|d|S)NzContrast RatiorzViewing Angle (L/R/U/D)z Viewing Anglez Key Features Certified)r:r;)rr viewing_angle key_featuress r_display_join_rowsr?vsvD yy!" .?O9PQRII78M o FG99^,L kLAB KrgroupsctDcgc] }||vs| }}|td|Dz }|Dcgc]}||s |||dc}Scc}wcc}w)Nc32K|]}|tvs |ywr)_CATEGORY_ORDER)rTcs rrVz_finalize..sDA1O+Cas )rr)rCsorted)r@rDordereds r _finalizerGs])9/QQ&[q/G9 vDDDDG8? M16!9F1I . MM: Ms A A  A A psref_rowc|r|jdsggdfS tj|d}i}i}t t }tD]9\}}|j|}|j||s+t|||||;tD]8\}}|j|}|j||s+t||||:t||||t||t|||t||jd} | r1t!| D]#} |j#dgj%| %t'||} | r1t)| D]#} |j#dgj%| %|j+D]>\}}||vr t,j|jdd}t|||||@t/|t/||jdfS#ttf$rggdfcYSwxYw) zRun at read time (not baked into the backfill), so it always reflects live outlet fields. Returns (specs_extra, reference_specs, reference_specs_url).r)N processorrrr6rr[)r:rcrrrrset_EXCLUDE_FNAMESraddrrrrrrrr!rr;r:r?r#_L2_FALLBACK_CATEGORYrG) rrHr)rrhandledrArrprocessor_matchr display_matchs rresolve_specs_for_laptoprRs IMM+62t|*.**Y{5K*L ')E*,I/"G'x e$ E  5(E5) D ( (x e$ E  uhu 5 ( vy%;Iu-vy%(&y&**[2IJO'8C   [" - 4 4S 99#9f5M%m4C   Y + 2 23 75 ") u G  (,,UYYt_jIuhui@ * U Yy19==3L LLS z "2t|sG33H  H r)@__doc__rrcrr urllib.parser^typingrrrrrbackend.configrbackend.db.connectionr backend.logging_configr r backend.scraper.apir rstr SpecFieldrr(rrGrkrvintrSpecRow SpecGrouprCr__annotations__rrLrNrcompilerrPatternrrrrrrrrrrr!r#r&r(r:r?rGrRrrrcs> 33#/5( cN  8C= S $DcNs#$t*#c9n1E#LSXd38n%=8csxS)^@T7U>D8c(;D8tCQTH~D8Z sCx. cN  "' d5c?#.' d5c?#4# '399&` `` ` T']" # ` Cg&' `  `8RZZ%'<=RZZ%w/RZZ RTT"G,RZZ'RZZ%rtt,m<RZZ"BDD):6RZZ=rttDlSRZZ rtt$&9:RZZ&-}=RZZ%rtt,j9RZZ BDD!5) 6d5S12 RZZ%}5RZZ"$$)RZZ BDD!5)RZZ'RZZ BDD!?3RZZ BDD!9-RZZ BDD!#34RZZ BDD!9- 6d5S12 CC  i  3  s  4TRY]HZC[  `d  J cNJCN#J T']" #JCg&' J  J4 \tCN'; \DdSZmI[D\ \ae \ StCH~ S$sI~2F StTWY]^eYfTfOg Slp S  c  c   S)^ 4 QT ZbcghkmphpcqZr B0 0QYZ]Q^0cklocp00 S#X 4= ,S,Xe_, A3A8C=A d3 >2DcNxX\]`be]eXfOg< d38n g Nd3W -.N4 ?N 2M cN2M'/S#X'?2M 4 ?DOXc] :;2Mr