o gح @sdZddlZddlZddlZddlmZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZddlmZddlmZddlmZddlmZddlmZdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%m&Z&m'Z'ddl(Z(dd l)m*Z*dd l(m+Z+d dl,m-Z-m.Z.m/Z/d dl0m1Z1Gddde2eZ3Gddde2eZ4Gddde2eZ5e5j6de5j7de5j8de5j9de5j:de5j;diZde=j?diZ@Gd d!d!e2eZAGd"d#d#e'd$d%ZBGd&d'd'e'd$d%ZCd(ZDd)ZEd*ZF   d?d+e!e2d,e!e2d-e!eGd.ee"eGe2fddffd/d0ZHddd$e=j>fd+e!e2d1e!e2d2eId3e=d.eGf d4d5ZJe5j8e=j>fd1e!eee2e2fd6e#e5e2fd3e#e=e2fd.e!e2fd7d8ZKGd9d:d:ZLGd;d<d<ZMGd=d>d>ZNdS)@z6 Classes for interacting with Salesforce Bulk 2.0 API N) OrderedDict)ThreadPoolExecutor)closing)Enum)partial)sleep) AnyAnyStrDict GeneratorListMutableMappingOptionalTupleUnion)Literal NotRequired TypedDict)chunked)Session)SalesforceBulkV2ExtractErrorSalesforceBulkV2LoadErrorSalesforceOperationError)call_salesforcec@s(eZdZdZdZdZdZdZdZdZ dS) Operationinsertupsertupdatedelete hardDeletequeryqueryAllN) __name__ __module__ __qualname__rrrr hard_deleter! query_allr(r(P/home/ubuntu/webapp/venv/lib/python3.10/site-packages/simple_salesforce/bulk2.pyr(src@$eZdZdZdZdZdZdZdZdS)JobStateOpenAbortedFailedUploadComplete InProgress JobCompleteN) r#r$r%openabortedfailedupload_complete in_progress job_completer(r(r(r)r+4r+c@r*)ColumnDelimiter BACKQUOTECARETCOMMAPIPE SEMICOLONTABN) r#r$r%r:r;r<r=r>r?r(r(r(r)r9?r8r9`^,|; c@seZdZdZdZdS) LineEndingLFCRLFN)r#r$r%rGrHr(r(r(r)rFTsrF z c@seZdZdZdZdZdS) ResultsType failedResultssuccessfulResultsunprocessedRecordsN)r#r$r%r4 successful unprocessedr(r(r(r)rJasrJc@seZdZUeed<eed<dS)QueryParameters maxRecordslocatorN)r#r$r%int__annotations__strr(r(r(r)rPis  rPF)totalc@s6eZdZUeed<eed<eeed<eeed<dS) QueryResultrRnumber_of_recordsrecordsfileN)r#r$r%rUrTrSrr(r(r(r)rWps  rWi@ iPfilenamerY max_recordsreturnc cs|r t|ddnt|dd}|rtj|nt|}|p|}t||}t|td}d}d}g} |rt|dd[} | } | D]7} |d7}|t | d7}||ksX||krs| rf|d| d | fV| g} d}t | d}qA| | qA| r|| d | fVWd d SWd d S1swYd S|d usJ|dd} |ddd D]7} |d7}|t | d7}||ks||kr| r|d| d | fV| g} d}t | d}q| | q| r|| d | fVd Sd S) zSplit a CSV file into chunks to avoid exceeding the Salesforce bulk 2.0 API limits. Arguments: * filename -- csv file * max_records -- the number of records per chunk, None for auto size T)r\ skip_header)datar_irutf-8encodingrN) _count_csvospathgetsizesys getsizeofminMAX_INGEST_JOB_FILE_SIZEr2readlinelenencodejoinappend splitlines) r\rYr] total_records csv_data_size _max_records max_bytes records_size bytes_sizebuffbisheaderliner(r(r) _split_csvsv "  r}r`r_ line_endingcCs|r"t|dd}tdd|D}Wdn1swYn|rz_count_csv..Nrcsrrr(rr(r(r)rrz(Either filename or data must be provided)r2sumrepr_line_ending_charrefinditer ValueError)r\r`r_r~rzcountpatr(r(r)res" recolumn_delimitercCsb|sdStdddd|DD}t}tj||||d}||D]}||q%|S)z*Converts list of dicts to CSV like object.Ncss|] }|D]}|VqqdSNr()rsir(r(r)rsz'_convert_dict_to_csv..cSsg|]}|qSr()keys)rdr(r(r) sz(_convert_dict_to_csv..) fieldnames delimiterlineterminator)setioStringIOcsv DictWriter writeheaderwriterowgetvalue)r`rr~rdict_to_csv_filewriterrowr(r(r)_convert_dict_to_csvs rc @sReZdZdZ  ddededeeeefdeefddZd ed d fd d Z dS)SFBulk2HandlerzBulk 2.0 API request handler Intermediate class which allows us to use commands, such as 'sf.bulk2.Contacts.insert(...)' This is really just a middle layer, whose sole purpose is to allow the above syntax N session_id bulk2_urlproxiessessioncCsF||_|pt|_||_|s|dur||j_dd|jdd|_dS)aInitialize the instance with the given parameters. Arguments: * session_id -- the session ID for authenticating to Salesforce * bulk2_url -- 2.0 API endpoint set in Salesforce instance * proxies -- the optional map of scheme to proxy server * session -- Custom requests session, created in calling code. This enables the use of requests Session features not otherwise exposed by simple_salesforce. Napplication/jsonzBearer 1) Content-Type Authorizationz X-PrettyPrint)rrequestsrrrrheaders)selfrrrrr(r(r)__init__s  zSFBulk2Handler.__init__namer^ SFBulk2TypecCst||j|j|jdS)N) object_namerrr)rrrr)rrr(r(r) __getattr__ s zSFBulk2Handler.__getattr__NN) r#r$r%__doc__rUrr rrrr(r(r(r)rs$  !rc @seZdZdZdZdZdZdZdedede eefd e fd d Z dCd e ede ede eeffddZ de ededefddZd ejejd fdede ededede edef ddZ dDdedededeejfdd Zdededefd!d"Zdedefd#d$Zdededefd%d&Zdeded'edefd(d)Zdededefd*d+Z d,e!de!fd-d.Z"d/e#fded0ed1e$de%fd2d3Z&d/e#d4fd5eded0ed1e$d6e$de%f d7d8Z' dEded9ed:e edd fd;d<Z(ded=edefd>d?Z) 4dFd@eded=ed6e$dd f dAdBZ*d S)G _Bulk2ClientzBulk 2.0 API clientrztext/csv; charset=UTF-8iQg@rrrrcCs||_||_||_||_dS)a Arguments: * object_name -- the name of the type of SObject this represents, e.g. `Lead` or `Contact` * bulk2_url -- 2.0 API endpoint set in Salesforce instance * headers -- bulk 2.0 API headers * session -- Custom requests session, created in calling code. This enables the use of requests Session features not otherwise exposed by simple_salesforce. N)rrrrrrrrrr(r(r)r4s z_Bulk2Client.__init__Nrequest_content_typeaccept_content_typer^cCs,t|j}|p |j|d<|p|j|d<|S)z$Get headers for bulk 2.0 API requestrACCEPT)copydeepcopyrJSON_CONTENT_TYPE)rrrrr(r(r) _get_headersKs z_Bulk2Client._get_headersjob_idis_querycCs8|sd}|r |jd}n|jd}|r|d|}|S)z"Construct bulk 2.0 API request URLrdr!ingest/)r)rrrurlr(r(r)_construct_request_urlVs  z#_Bulk2Client._construct_request_url operationr!rr~external_id_fieldc Cs|||d}|r ||d<|tjtjfv}|d|}|r/||j|j} |s*td||d<n||j|j} |j|d<d|d<t |d |j | t j |d d d } | j t d S)asCreate job Arguments: * operation -- Bulk operation to be performed by job * query -- SOQL query to be performed by job * column_delimiter -- The column delimiter used for CSV job data * line_ending -- The line ending used for CSV job data * external_id_field -- The external ID field in the object being updated )rcolumnDelimiter lineEndingexternalIdFieldNameNz Query is required for query jobsr!objectCSV contentTypePOSTF allow_nanrmethodrrr`object_pairs_hook)rr!r'rrrCSV_CONTENT_TYPErrrrjsondumpsr) rrr!rr~rpayloadrrrresultr(r(r) create_jobgsH   z_Bulk2Client.create_job?waitc Cstjtj|jd}|rtjntj}d}d}t|tj|kri|||}|d}|tj tj tj fvrL|tj krJ| dpB|} t d| |S||jkr^|t|d}|d7}t|tj|ks#t d |) z"Wait for job completion or timeout)secondsgrstate errorMessagezJob failure. Response content: g@@rzJob timeout. Job status: )datetimenow timedeltaDEFAULT_WAIT_TIMEOUT_SECONDSr+r6r2rget_jobr7r3r4getrMAX_CHECK_INTERVAL_SECONDSmathexp) rrrrexpiration_time job_status delay_timeout delay_cntjob_info error_messager(r(r) wait_for_jobs<   z_Bulk2Client.wait_for_jobcCs|||tjS)zAbort query/ingest job)_set_job_stater+r3)rrrr(r(r) abort_jobsz_Bulk2Client.abort_jobcCs||dtjS)zClose ingest jobF)rr+r5)rrr(r(r) close_jobs z_Bulk2Client.close_jobcCs2|||}|}t|d|j|d}|jtdS)zDelete query/ingest jobDELETErrrrr)rrrrrr)rrrrrrr(r(r) delete_jobs z_Bulk2Client.delete_jobrc CsF|||}|}d|i}t|d|j|tj|ddd}|jtdS)z Set job staterPATCHFrrr)rrrrrrr)rrrrrrrrr(r(r)rs  z_Bulk2Client._set_job_statecCs,|||}t|d|j|jd}|jtdS)z Get job infoGETrr)rrrrrr)rrrrrr(r(r)rs z_Bulk2Client.get_jobbcCs4t|tr |ddSt|tr|ddStd)zB https://github.com/airbytehq/airbyte/issues/8300 rdzExpected str or bytes) isinstancerUreplacebytes TypeError)rrr(r(r)filter_null_bytessz_Bulk2Client.filter_null_bytesrdrRr]c Cs||dd}d|i}|r|dkr||d<||j|j}t|d|j||d}|jdd }|dkr5d }t|jd }||| |j d d S) Get results for a query jobT/resultsrQnullrRr)rrrrparamsSforce-LocatorrdSforce-NumberOfRecordsra)rRrXrY) rrrrrrrrrSrcontentdecode) rrrRr]rrrrrXr(r(r)get_query_results)s< z_Bulk2Client.get_query_resultsrg chunk_sizec CsTtj|s td|||dd}d|i}|r#|dkr#||d<||j|j}tt |d|j ||ddh} t j d |d d d N} | j d d}|dkrPd}t| j d} | j|dD] } | || q]tj| jr|| | jdWdWdStd| jd|d1swYWddS1swYdS)rPath does not exist: TrrQrrRr)rrrrrstreamwbz.csvF)dirsuffixrrrdrr )rRrXrZN7The IO/Error occured while verifying binary data. File  doesn't exist, url: , )rfrgexistsrrrrrrrrtempfileNamedTemporaryFilerrrS iter_contentwriterisfiler) rrgrrRr]r rrrrbosrXchunkr(r(r)download_job_dataOsr    4z_Bulk2Client.download_job_datar` content_urlcCs|stdt|d}|tkrtd|d|p"||dd}||j|j}t|d|j ||dd}|j t j krLtd |j d |j d S) zUpload job dataz Data is required for ingest jobsraz Data size z7 exceeds the max file size accepted by Bulk V2 (100 MB)Fz/batchesPUTrz&Failed to upload job data. Error Code z. Response content: N)rrnrorlrrrrrr status_codehttpCREATEDrr )rrr`r data_sizerrrr(r(r)upload_job_datas@   z_Bulk2Client.upload_job_data results_typecCs<||dd|}||j|j}t|d|j|d}|jS)zGet record resultsFrrr)rrrrrrtext)rrr&rrrr(r(r)get_ingest_resultss&z_Bulk2Client.get_ingest_resultsrZc Cs||dd|}||j|j}tt|d|j|d.}t|d}|j|dD] } | | | q*Wdn1s?wYWdn1sNwYt j |sdtd|d |d dS) z!Download record results to a fileFrrrrrNrrr)rrrrrrrr2rrrrfrgrr) rrZrr&r rrrrrr(r(r)download_ingest_resultssP  z$_Bulk2Client.download_ingest_resultsr)rr)r )+r#r$r%rrrrrrUr rrrrboolrr9r<rFrGrrrfloatrr+r7rrrrrrr rDEFAULT_QUERY_PAGE_SIZErSrWr rr%r(r)r(r(r(r)r+s4       > &         * A ) rc@s"eZdZdZdededeeefdefddZej e j dd fd e d e eeeeffd ed e deededeeeffddZdddej e j ddd fd e deedeedeed ed e deedededeeeeffddZdddej e j dd fdeedeeeeefdeed ed e deededeeeeffddZddddej e j d fdeedeeeeefdeeded ed e dedeeeeffddZddddej e j d fdeedeeeeefdedeed ed e dedeeeeffdd Zdddej e j d fdeedeeeeefdeed ed e dedeeeeffd!d"Zdddej e j d fdeedeeeeefdeed ed e dedeeeeffd#d$Zeej e j d fd%ed&ed ed e dedee eefddff d'd(Zeej e j d fd%ed&ed ed e dedeeddff d)d*Zeej e j d fd%ed+ed&ed ed e dedeefd,d-Z d;d.ed/ed0eedefd1d2Z d;d.ed0eedefd3d4Z d;d.ed0eedefd5d6Z! d;d.ed0eedefd7d8Z" d;d.ed0eedeeee#ffd9d:Z$dS)r?r@rFrGrHrrJrPrWrlrAr,rSr}r*rerrrrr(r(r(r)s       ,            F    4F