Hastructure
a powerful and extensible open-source cashflow/analytics engine for structured finance world.
support customized waterfall distribution, trigger etc
shipped with RESTful interface as well as a docker image.
HastructureAfter a few months in auto-pilot mode, the web app can now auto-pull from the data source, digest with an LLM, and normalize into the database. The UI shows the data in an almost real-time fashion (only 15 minutes, end to end!).
On the tech stack, Airflow is just too heavy, and APScheduler is a good replacement.
I just subscribed to an AI/LLM service, and it does a great job assisting with web application development as well as the Hastructure engine.
Now new features can be driven fairly quickly with the help of a coding agent! That makes it possible to speed up work on legacy bugs and refactoring and, more importantly, the verbose unit tests.
I'm quite confident that, with the help of AI/LLM, Hastructure and absbox will enter a new era. Items I'm thinking of:
protobuf, which allows interaction with more languages. It used to be tedious to map, but it will become possible; AI will take over those responsibilities.MCP on the data feed, which lowers the barrier for users to access it (no more RESTful API).skills. Actually, it is quite a low-hanging fruit! That might help users pick up absbox even quicker. absbox-skillRecently I have been using LLMs extensively to support the technical infrastructure for structured finance and asset securitization. The results have been surprisingly good. We are now getting strong performance extracting bond attributes, waterfall allocation rules, and triggers or credit enhancements.
With these technologies and Airflow workflows, the SQLite database has accumulated a lot of dataāapproaching 1 GBāso we are considering migrating to PostgreSQL.
FastAPI has proven very useful for building a deal library and a data-feed API service for absbox.
We have collected many requirements related to non-performing assets. Significant enhancements are planned for Q1 2026. Q1 priorities are:
I've been working on this for 2 months! Before releasing this I spent many days thinking about how this can be different from other vendors: is it going to be faster? More machine-readable?
This product will cover the entire Chinese inter-bank market, with full coverage of deals that disclose trustee reports.
The greatest difficulty in parsing trustee reports in the China market is diversity: different issuers use various formats and styles.
/ may be OCR'd as the number 1. may be OCR'd as ,7.5% while others use 0.075 or just 7.5The solution is to design a DSL to describe the report.
(percentage rate).balance value.Given these annotations, the logic I wrote understands them and performs the conversion.
7.5% will be converted to 0.075 for consistency./ will be read as 0 if it is a balance.This reduces a lot of complexity, and supports multilingual inputs.
I tried a bit of LLM assistance and it is amazing: instead of writing hundreds of regular expressions (which I'm experienced at), a simple question to the LLM API will often get you what you need.
With the help of structured output helpers like instructor, productivity is greatly improved; the only concern is cost, so I will adopt a multi-vendor strategy to slow cash outflow.
I'm still thinking about the tech stack; I want it to be simple, lightweight, and scalable. I'm leaning towards HTMX+Alpine while gathering more feedback on nicegui.
One certainty is that I'm starting to build the web API with FastAPI and will replace the legacy Flask-based deal library API.
It's worthwhile to send out the heartbeat to the community to keep confidence afloat; I'll try to note what happened in Hastructure & absbox on a quarterly basis.
I've included code-based regression tests in the test-suite. The old JSON-based cashflow comparisons will be sunset because they produce too much noise.
It's great: marimo can be served as a notebook for exploratory research, as a script, or even as an App!
Adopting marimo will replace many notebooks in the docs and will be used in the backend deal library (which will phase out htmx-alpine). This lets me be relieved from some web development tasks.
The pilot development will rebuild this website (absbox.org).
Even with help from Stack and GHCup, there is still friction when installing different versions and managing dependencies. I chose Nix as the final solution and climbed the learning curve; it worked well after I installed it on two laptops running NixOS and resolved a couple of driver issues.
The benefits:
Enterprise users feel more comfortable building from source if they choose Nix as the package manager; now it's a one-liner to build Hastructure.
I'm more relaxed about updating dependencies for Hastructure.
As of Aug 1, Hastructure is on stackage and should be easy to install via stack; hopefully it will appear in nixpkgs as well.
Happy New Year 2025!
2024-12-30 -> 2025-01-08 in Melbourne and Sydney
Early booking via Email to support@asset-backed.org
sudo xcode-select -s /Library/Developer/CommandLineTools) before building, working around the ld/SDK mismatch (tapi error: malformed file, arm64e.x1-macos) that broke linking on the macos-26-arm64 image 20261013.0424.1 (actions/runner-images#14940). 0.52.13 was not published because of this CI failure.aeson-gadt-th dependency is gone, upper bounds widened (aeson <2.4, time <1.17, containers <0.9, base-compat <0.16, hashable <1.6, parallel <3.4, servant-openapi3 <2.2) and unused dependencies pruned from every component (library, executable, test suite); verified with all 531 tests on GHC 9.10.3 and 9.12.3haskell:9.12.4-slim-bookworm, Stack moves to LTS 24.63, and build.nix / release.nix pin nixos-26.05 (stale generated nix files removed)v* tag now publishes the release to Hackage automatically (previously dry-run only)-t flags; "Publish Multi-Arch Manifest" recovery workflow added (PR #322 H2)config.yml field documentation; Features section made collapsibleLib.prorataFactors clamps negative balances (treated as zero) instead of producing negative allocations (PR #322 L3)UT.LedgerTest tracked in the cabal file; Lib.periodsBetween removal documented (PR #322 L2/L3)sumTsCF now takes NonEmpty TsRow (source-level breaking for library consumers) instead of throwing on an empty group; the dead, unexported partial sumTs (foldr1) is removed; firstDate returns the frame's begin date for an empty frame (epocDate for DummyCF) instead of failing; emptyTsRow and viewTsRow gained the missing CashFlow clause so extendCashFlow / buildCollectedCF zero-row synthesis no longer crash on generic rows; buildCollectedCF lost its unreachable error clause. aggTsByDates keeps its signature and behavior (empty groups are skipped via NE.nonEmpty).m >= 1 + 1/r) instead of dividing by zero (Rational exception) or emitting negative SMM; 100% rate edge covered. The remaining over-reporting when SMM > 1 before exhaustion is tracked in #358.{"Left":"ā¦"} and application/json) instead of a plain-text body with no content type, matching the in-band errors and the #344 boundary (PR #322 review M1).Either results to WHNF and every value of the multi-run response maps via Util.forceMapValues ā so engine exceptions surface before Warp starts the response and are rendered by the JSON error boundary. Previously the response body was only forced while Warp was already streaming it, so error/partial-function failures inside computed responses aborted the connection (empty reply) instead of returning the JSON envelope. Deep laziness inside response payloads may still surface during streaming (see forceMapValues).RunDealReq / RunPoolReq / RunAssetReq / RootFindReq: a request decoded with the wrong constructor for the routed endpoint crashed with a pattern-match failure (now rendered as an opaque engine error: 500 by the #344 boundary). Every handler is now total. A misrouted request to a single-run endpoint (runDeal, runPool, runAsset) returns the reason in-band as Left "<endpoint>: expected <Constructor>"; map-valued multi-run endpoints (whose response type has no error slot) answer HTTP 400 with the same message, uniform with the #345 run-cap rejections. wrapRunAsset rejects non-PoolLevel assumption shapes (per-pool / per-index / obligor strategies and pricing without assumptions) in-band.runByRootFinder no longer crashes on the legacy FirstLossReq / MaxSpreadToFaceReq request shapes (kept on the wire but unsupported; absbox composes RootFinderReq for both runFirstLoss and runRootFinder); a stress tweak (stressDefault / stressPrepayment) without pool assumptions is rejected with a descriptive error instead of crashing; the BalanceFormula stop condition is rejected as not implemented yet. The executable now compiles with -Wincomplete-patterns -Werror=incomplete-patterns, so any future non-exhaustive match in the API layer fails the build (the library is unchanged); note locally cabal needs one clean rebuild to pick up the new flags.
error calls, partial accessors, pattern failures) now answer 500 with the same envelope as in-band failures ({"Left":"engine error: ā¦"}) and the CORS header, instead of Warp's opaque plain-text 500; server-side exception logging is unchangedmaxScenario / maxDealNum limits declared in config.yml were dead configuration: the engine never read them and scenario/deal grids were unbounded (DoS via large runByCombo cartesian products). Each multi-run endpoint (runPoolByScenarios, runDealByScenarios, runMultiDeals, runDealByRunScenarios, runByCombo) now enforces a per-request total-run cap, default 1000, configurable per endpoint via the new maxRunNum.<endpoint> keys in config.yml; runByCombo sizes the cartesian product before materialising it, so oversized grids are rejected without allocation. Oversized requests are rejected with HTTP 400 naming the endpoint, the configured limit and the key to raise. The in-flight worker cap is now configurable via workerNum in config.yml (precedence: HASTRUCTURE_WORKERS env > workerNum > RTS -N); the internal unsafePerformIO global is gone.config.yml, a missing port or a non-positive maxRunNum.* value now aborts startup with a descriptive error instead of silently falling back to defaults; the startup log prints the effective port, worker count and per-endpoint limits. The unused maxScenario/maxDealNum keys are removed from config.yml.
TxnComment JSON codec is now total and self-consistent: malformed, legacy or wrong-arity comment strings fail with a descriptive Aeson parse error instead of throwing (head/!!/read and the greedy regex are gone); the encoder emits canonical forms for BankInt (<BankInt:>), PoolInflow (<Pool:ids,Source>), LiquidationSupportInt (<SupportExp:b1,b2>) and BookLedgerBy (<BookLedger:dir,name>), all of which now round-trip; multi-bond list comments decode to their elements; TxnComments arrays can be decoded; Read PoolId no longer throws and accepts what Show PoolId produces
Paths_Hastructure) instead of a hardcoded string, so /version and the OpenAPI schema always match the built packageValidation module and de-duplicate pragmas / imports in Deal.DealValidationpriceBonds) no longer crashes when a requested bond name does not exist; it returns a descriptive errormflowBorrowerNum / mflowPrepaymentPenalty no longer fail with undefined on non-mortgage cashflow rowsWeekly / BiWeekly / Daily periods are now handled by periodRateFromAnnualRate, afterNPeriod, genDates and nextDate (previously a pattern-match failure or a silently non-advancing schedule)Map.!: not foundfoldl1: empty list when a deal has no collection ruleshead / last usages across the engine (curves, statements, date generation, asset projections, bond WAL, deal dates and cashflow aggregation) so empty input yields a defined result or a typed error instead of a runtime exceptionDefaultByAmt pro-rata allocation coverage with edge-case unit tests (cent rounding, zero-balance assets, empty pool, multi-period weights, negative totals) (#321)PrepaymentABS, an ABS / absolute prepayment model driven by the asset's seasoning (months elapsed since origin). Supported for Monthly assets only.DefaultByAmt is now allocated across multiple assets proportionally to their current balance.Lib.periodsBetween is removed in favor of Lib.monthsBetween :: Date -> Date -> Integer (whole calendar months between two dates; partial months clipped, so 2021-01-31 -> 2021-02-01 is 0 months; negative when the second date is earlier).ConstTrue / ConstFalse for bool type formulaIrrOfBond as a formula, which return the irr of the bondIsAnyOutstanding as formula: return True if any of the bond is outstandingNEW: add PoolAccruedInterest as a formula: return the accural amount of the pool
ENHANCEMENT: allow negative amount when calculating AmountRequiredForIRR
activeBondNumberlinear to Hermite wayprepayment default recovery .Buy BondProjectedByFactor and ProjectedCashflow .name is missing from the deal objectcredit support for liquidity suppport.stopBy in run assumption ,which stop deal run by a list of Conditionasset level cashflow ,with a toggleUn-Used Pool cashflowtweaksHackageLease deal for Financial reportstweak: Stress Prepayment stop: Bond Principal Loss Bond Interest Losstweak: Balance Split and stop: Bond Met Target IRRleaseEndType assumptions: Earlier Later which will end the lease projection base on two input End date and extention times.new bond rate type in trigger effects. Now bond rate type can be changed during the projection.root finder endpoint and signature. In the long term, the refactor of signature lays down fundation for deal structuring domain, now it would be easy to implement all kinds of structuring features.BaseByVec for vector-based rental changeMaxSpread feature for structuring stage: get max possible bond coupon rate !stack to cabal as build toolDList to trigger logDouble Decline Balance in 🐞FixedAssetLeasing asset type
Default assumptionPeriod-based rental ,in addition to Day-based rental calculationList with DList.inspection ,expose IsOutstanding HasPassedMaturity in PrePricing/IRR, error when holding position is too smallinterest start date for bonds if it is not modeled. In PreClosing status, engine will use closing date as bond interest begin date ; In Non-PreClosing status, it defaults to use last waterfall distribution date as bond interest begin date.DealDates : FixInterval, CustomDates and PatternInterval. Since all these can be replace by new GenericDates in type DateDespNo last interest accure day will begin accrue interest from closing date if the deal is in PreClosing mode, while the bond will use last bond day otherwise.IsPaidOff now can be queried in inspection formulaPAC PAC Anchor to BondGroup, now BondGroup is Map String L.Bond (Maybe PrinType)bondTargetBalance to query target amortized balancePAC Anchor which is same to PAC except that the balance schedule will be ineffective if Anchor Bonds are paid off.BalRef which bond will accrue its interest by a Formula, which is being used to model IO bondFirstLoss ,the stress will be applied to revolving assumption as wellinterest accrued in bond pricing resultCalcBondInt now only accepts a list of bond nameslease has been refactoredholding a bond, hold and sell a bond, or buy a bond.Pricing to expose error message.GT to > and others as well.bond paid periods and pool collection periods for preClosing deal.byTerm assumption on Installmentroot.finderPeriodBased rate curve or balance curve in Pre, i.e. easy to build default rate trigger in structuring stageFirstLoss as new endpoint, which will stress on Default assumption till 0.01 loss on input tranche.byTerm, which vector curves are being applied via term of the assets.Multi-thread on pool cashflow projectionconvexity on bond/assetPSA for Monthly mortgageBondPaidPeriod PoolCollectedPeriod indexcalcPmt to boost 15x performance for mortgage cashflow projection. ScheduleByIndex for bondsfundWith shall increase the bond balanceMulti Interest Bond which used to model in bond with step up feature ( sub ordinated interest) in European totalFunding for bond,which records all funding amount of bond.AmountRequiredForIRR for bond,which return the amount to be paid out make bond met the target IRR.Rate Swap , the notional can be set by Formulasupport, it has option to book ledger with both credit and debit direction.payPrinBySeq was not paying out principal.totalFunded for bond with extra funding amountFundBond, which records a time series funding amount for a single bond.support action, now user can book on Credit or Debit sidepayPrinBySeq was not workinglimit always return positive ,otherwise engine will throw errorchangeStatus in waterfall, with optional Pre as condition to trigger the status changeledgerBalanceBy, which return either Credit or Debit balance of a ledgerPDL book type.dates to be testedTree from a TablewriteoffBySeq which write a list of bonds by sequence.transferMultiple, with one action transfer multiple account to single accountinclude or not include accrued interest.runPool cashflow order and add UT weighted average formula.annualized rate fee type with formula bondbalance on bondGroupissue bond which allow funding by issuing new bonds during cashflow projection.projectScheduleFlow which can be divided projected cashflow with fix portion and float portions. The interest from the float portion will be affected by interest rate assumption.bondRate/bondWaRate on bondGroupformula will return inf if a divide with zero instead of just throw exceptionfinancial reports was failing because it can't access to interest due on bond group.bond groups limit on payFee was not working with duePcttransaction statement for triggerstrigger to run waterfall actionsresult log used to be doubled each pool collection periodpayPrinResidual will use all cash from account regardless principal due of bonds, which may caused negative balance of bonds( cash of account > principal due of bond)bondGroup, which group bonds and pay with prorata/sequential/by coupon rate/by maturity/by start datebegin balance/accure interest/as of date for cashflow frameinterest arrears interest over interest on bond cashflowinterest over interest settings on bonds and expose interest over interest interest due flowweekday <n> in the date patternweekly /biweekly in Periodfirst N period without Fee feature to model cashflow of type Installment Combination Type formula (via patching dates)commentoriginalBondBalance,BondDuePrinCalcBondPrin,PayPrinWithDuePoolFactor* between formulasduration for asset pricing (curve only)ReceivableRecoveryByDays to Receivable ,which describes recovery cash received after default.leaselts-18.22 to lts-22.6Apple silicon chip ! Happy hacking Mac users !Mortgage type cashflowReceivable which represent a invoice factored,trading receivableDefaultAtEnd assumption, which assumes asset default at last payment(For Receivable)fundWith which will increate the balance of bond and deposit cash to account.writeOff which will write off balance of bond via a formulapassMaturity which True if bonds has passed their expected maturity/pay off date.Not as composite boolean test.OAS pricing assumption, which return OAS spread given input scenarios.NO_FirstN as type of Mortgage which implies no payment for first N period and interest due will be capitalized.IO_FirstN as type of Mortgage which implies no principal payment for first N period.Make Whole Feature, which allow user to set a resec deal, which allow to use bonds as underlying assets and allow user to set assumption on underlying deals.payIntBySeq which pay interest to bonds sequentially with optional limitNothing to trigger effectspayFeeBySeq to which pay a list of fees sequentially with optional limitrate and balance to calcDueInt action in waterfall.present value on schedule pool cashflow, which enable Yield Maitenance Overcollaterisaztion supportsFixAsset type , which yield cashflow given a capacity and assumption called utilization rate curve. The new asset type is applicable to Hotel booking/EV Charge station/Solar Panel/Wind Power type.RateCap which yield cash if rateCurve is higher than a strike rateaccruedInterest field in pool stats, which will be deducted from pool cash flow payPrinBySeq in waterfall action, now user can pay prin bond via a simple list.fireTrigger which mannualy fire a trigger at point of projectiontotalCash will aggregate all pool cash fieldpayInt now accept a limit which constrain how much interset to be paid via a formulabookBy a ledger via formulaI_P to Mortgage type ,which models Buy To Let type mortgage( interest only and principal at last period)Lens and code clean upStepUp out of interest part of bond.runDealdefault by amount assumption, which enable user to set a total amount of default alongside with a vector.Cumulative Stats ( cumulative default/delinq/loss/prepayment/principal/recovery)inspect in waterfall action to observe variables during a waterfall executionstepup now accpet a pre instead of a date to switch rateissuance balance for PreClosing Dealpre-run check and post-run check
Ongoing Deal shall have a IssuanceBalance value in Poolprepay penalty when using stepDownLoanpreClosing stagedefault/delinq/loss status map when projecting cashflowhaircut as extra stress projecting mortgage called deal status, which will be set when deal was triggered with a clean up call assumptionrunAsset endpointdeal status as well as trigger statusrampUp deal statuscutoff date to closing dateperformance assumptiondelinquency projection on mortgage as well as schedule mortgage cashflowinspect fieldTrigger from list into a map with a nameCumulatiePoolDefaultedRateTill to query default rate as of collection period N , then support query last one,last two default rates in the past as a rolling basis..queryBool with test logic of any or which will test all predicates or any predicates are/is satisfied. With new included aforementioned formula above, the engine can have a predicate like last 2 period cumulative defaulte rates are all lower than 5%, any last 2 period cumulative defaulte rates is higher than 5% LedgerTxnAmt , allow user to query transaction amount for a ledger by commentAbs in formula , which will get absolute value of another formulapayInt and payFee which includes extraSupport from either another account or liquidation provider, with option to book PDL draw on ledgerCumulative Net Loss Cumulative Net Loss Ratio Bond Rate Bond Weight Average Rate in formulaAvg in formula ,which can calculate average value from a list of deal stats.RefRate in bond , now bond can be setup interest rate which reference to a value of deal , could be like 100% of Pool WAC coupon , or average of bond rate of bonds etc.liquidity provider interest swap to balance sheet repotis_most_senior_bondPoolCurCollection returns target pool source balance in last collected periodtarget reserve amountreserve account excess/gap to formulastep up coupon by date ,which pertains to Euro dealsHackageStep Up By Date /Cap/Floor coupon type for bondPrepay Penalty attribute on Mortgage, penalty types includes:
rate0 before term N and rate1 after term Nterm Nterm Nrate0 by step of rate112 periods with Pct of Rate0 , next 12 periods with Pct of Rate1liquidity provider
valid date of the agreementcalcAndPay action for feeNO_IE type to generate dates vectorrecurr type of feeTargetBalanceFee, which due amount = <formula 1> - <formula 2>comment as a filtercumulative pool on recoveries principal interest prepayment beg balance on poolcumulative pool recoveriesfactor in querybookBy actionpayInt action and accrueInt actionZ-spread calculationdeal.hs , break down code logic into seperate files.ledgers to accomodate PDL feature (Principal Deficiency Ledger)rounding on deal stats, which rounds interest rate change by a factor of fix amount ,or pay principal on balance by a factor of fix amount.runDate endpoint as sandbox for user to play with <datePattern>floorAndCap formula to set upper or lower bound of formula valuepct and annual type of feelimit for revolving buy actiondefault waterfall Pre now support comparing with a balance type formula ,not limited to a balance numberexclude dates and offset by days InspectBalanceSheet Report and Cashflow Report, user can query them via set flags in assumptionCashflow ReportStep-Up coupon feature DatePattern to annotate reset date for floater bondsInstallmentDefaultedRecovery assumption for defaulted assets.AdjustRateMortgage with assumption:
If user prefers to accelerate the onboarding process and avoid back&forth research on the deployment and integration, there are couple commercial options available:
a production ready deal life cycle deal management toolkit in Python
Accelerating the integration process by
Chinese Structured Finance Market Data Feed
Email: support@asset-backed.org