This document sets out how a raw corpus of Nigerian news articles was turned into three analysable products: a structured incident datatable, a behavioural sequence model, and a longitudinal record of media coverage. Each processing stage is described in plain terms and then given as reproducible Python. The accompanying script, k4r_pipeline.py, runs the whole procedure on any similarly shaped dataset. Throughout, the guiding principle is that the outputs describe media-reported kidnapping, not the full incidence of the crime, and that values which cannot be supported from the text are left blank rather than estimated.
The input is a spreadsheet with one row per news article. The only fields required are the article's full text, its publication date and its source; a unique identifier and a URL are used where present.
The pipeline is written against a small configuration block so that it can be pointed at other corpora without editing the logic. Column names, the year range, and the two scoring thresholds are all set in one place.
# CONFIG — set once, then run unchanged CONFIG = { "col_text": "full_text", "col_date": "published_date", "col_source": "source_name", "col_url": "source_url", "year_min": 2014, "year_max": 2026, "incident_score_min": 7, # structured datatable "relevance_score_min": 3, # longitudinal corpus }
Behind the incident table sits a four-stage collection system that turns live news into structured records. The reproducible script documented later in this paper implements the extraction and analysis stages; the full production system adds continuous discovery, scraping, a relational store and event linking.
The system is designed so that each stage writes to a database and can be re-run independently, which makes the corpus maintainable as a living resource rather than a one-off scrape.
Two collectors feed the corpus. A GDELT collector filters the Global Database of Events, Language and Tone to records geotagged to Nigeria whose URLs match an allowlist of nineteen Nigerian newspaper domains (Punch, Vanguard, Daily Trust, The Nation, Premium Times, Sahara Reporters and others), with historical coverage reconstructed by direct file addressing where GDELT's master list is thin. An RSS collector polls the same mastheads continuously to capture reports before they reach GDELT. Domain filtering at ingest is deliberate: without it, international wires would dominate a Nigerian security picture. Article bodies are then extracted with Trafilatura, which strips navigation and boilerplate from news HTML, with site-specific rules for awkward layouts and a minimum length filter to drop paywalled stubs.
Each scraped article is passed to a large language model with a structured prompt that first decides whether the article describes a kidnapping and, if so, classifies its dominant narrative as KIDNAP, RELEASE, UPDATE or DEATH, then extracts event-level fields (date, state, LGA, location type, victim and perpetrator counts, ransom demanded and paid, release status, response actors) and victim-level fields (name, age band, gender, occupation, residence, survival). The report type is produced in the same call because it governs how the record is later linked. Two backends are supported: Anthropic's Claude (Fable 5) as the default, which in testing produced the most consistent structured output at an 87% success rate, and a locally hosted open model via Ollama (around 74%) for development and for text that cannot leave the machine. Every extraction carries a confidence score; records below 0.7 are routed to a review queue rather than the main table, and periodic audits of high-confidence records estimate the false-positive rate.
Raw model output varies in wording, so a cleaning stage maps states, perpetrator groups, release outcomes, location types, gender and age to controlled vocabularies. A single real-world kidnapping usually generates several articles over time — an initial report, updates, then a release or death — so an event linker groups them. For each release, death or update record it scores candidate kidnap records on state (a hard requirement), fuzzy victim-name overlap, victim-count agreement and temporal proximity, accepting a link only above a minimum combined score with at least two independent signals in agreement. Linked records share an event identifier and yield a duration and a final outcome, which is what turns article-level rows into incident-level events. This same clustering logic is what the reproducible pipeline uses to derive the spread and persistence metrics.
k4r_pipeline.py is a portable, dependency-light reimplementation of stages three and four that runs on a spreadsheet of article text, so the extraction, normalisation and linking logic can be reproduced and audited without standing up the full infrastructure. The scoring thresholds, controlled vocabularies and confidence handling are shared across both.A keyword filter first restricts the corpus to articles that mention kidnapping, abduction, ransom or hostage-taking. Keyword matching alone, however, also catches opinion columns, lectures and legal commentary that discuss the subject without reporting an event. To separate genuine incident reports from commentary, each article is given a small additive score: points are awarded for concrete abduction verbs, named perpetrator types, ransom language, an outcome, the presence of security actors, a Nigerian state name and a monetary figure; points are deducted for commentary markers such as first-person opinion or the words lecture and editorial.
Two thresholds are then applied. Articles scoring at least three form the broad corpus used for the longitudinal media metrics; articles scoring at least seven form the higher-signal set used for structured, per-incident extraction.
def incident_score(text): t = text.lower(); s = 0 if re.search(r'\b(kidnapp?ed|abducted|whisked away|seized)\b', t): s += 2 if re.search(r'\bgunmen|bandits|kidnappers\b', t): s += 1 if re.search(r'\bransom\b', t): s += 1 if re.search(r'\b(released|freed|rescued|found dead|killed)\b', t): s += 1 if re.search(r'\b(police|command|army|troops)\b', t): s += 1 if any(st.lower() in t for st in NIG_STATES): s += 1 if re.search(r'\bn[\d,]+ ?(million|billion)?\b|₦', t): s += 1 if re.search(r'\b(in my view|lecture|editorial|columnist)\b', t): s -= 3 return s
Scoring is deliberately transparent and adjustable. Raising the incident threshold trades recall for precision; the value of seven was chosen because it reliably admits event reports while excluding reflective pieces.
Each high-signal article is reduced to one row of structured fields, following the reference schema. Fields fall into three tiers by how reliably they can be read from free text.
| Field | Method | Reliability |
|---|---|---|
| Kidnap State | First state name co-occurring within a window of an abduction verb, else first state mentioned | High (later corrected against the geocoder) |
| Location Type | Site keyword (road, highway, school, farm, home, church…) nearest an abduction verb | High |
| Location feature | Immediate setting; Car for road, highway and travel cues | High |
| Ransom demanded / paid | Demand and payment cues with amount parsed from N-figures and words | High for presence; amounts where explicit |
| Release / Release Status | Release, rescue and escape versus killing and body-recovery cues → Alive / Deceased | High |
| Total Kidnapped | Largest credible victim count in the text; defaults to one | Medium |
| Kidnap Date | Explicit dates near the abduction verb, standardised to Month DD, YYYY; year inferred from publication when absent | Medium |
| Victim's Name | Honorific-plus-name only in explicit victim context; officials, politicians, places and institutions rejected | Low; verify |
Two design choices deserve emphasis. First, location and state are resolved by proximity to the abduction verb rather than first mention, so that incidental references (a victim "returning from school", a quoted "Catholic priest") do not misclassify the site. Second, the victim-name reader is deliberately precision-first: it fires only inside an explicit victim context and rejects a blocklist of public figures, place names and institutions, leaving the field blank when no confident person-name is present.
def location_type(t): # choose the site keyword closest to an abduction verb best, best_d = None, 10**9 for vm in re.finditer(VERB, t.lower()): win = t.lower()[max(0, vm.start()-200): vm.end()+200] for pat, label in LOC_PATTERNS: m = re.search(pat, win) if m and abs(win_pos - vm.start()) < best_d: best_d, best = ..., label return best
Coordinates are not read from the text; they are generated from the place names it contains. The procedure has three steps.
osm_corrected, osm_confirmed or text_inferred.def geocode(place, state, cache): q = [f"{place}, {state} State, Nigeria", f"{place}, Nigeria"] for query in q: r = requests.get("https://nominatim.openstreetmap.org/search", params={"q": query, "format": "json", "limit": 1, "countrycodes": "ng"}, headers={"User-Agent": UA}, timeout=20) time.sleep(1.0) # respect the usage policy if r.json(): d = r.json()[0] osm_state = parse_state(d["display_name"]) # authoritative return {"lat": d["lat"], "lon": d["lon"], "osm_state": osm_state}
osm_confirmed tier should be preferred over place-only matches for close work.Named victims rarely appear in enough of the corpus to support a gender breakdown on names alone. Because Nigerian mass abductions are frequently gendered at the group level, for example the abduction of schoolgirls or of schoolboys, gender is instead inferred for each incident from the balance of gendered language in the report. Female and male markers are counted, and the incident is labelled Female, Male or Mixed according to their ratio, or Unknown when neither is present.
def gender_incident(t): f = len(re.findall(r'\b(schoolgirls?|girls?|women|female|mother|wife|mrs|she|her)\b', t)) m = len(re.findall(r'\b(schoolboys?|boys?|men|male|father|husband|mr|he|his)\b', t)) if f == 0 and m == 0: return 'Unknown' ratio = f / (f + m) return 'Female' if ratio >= .62 else 'Male' if ratio <= .38 else 'Mixed'
This is a corpus-level signal, not a determination about any individual. It is reported as an inferred distribution and should be read as such.
The behavioural model asks a different question from the datatable: not what an incident contained, but in what order its behaviours occurred. A codebook of behaviours, each with a chronological stage and an actor, is matched against every report. The behaviours present in an incident are then assembled into a sequence.
Here a specific problem of news writing must be handled. Reporting follows an inverted pyramid: it leads with the outcome and then recounts the event, so ordering behaviours by their position in the text mixes narrative order with event order and produces impossible transitions such as a killing preceding an abduction. To recover event order at scale, behaviours are ordered by the canonical stage of the kidnapping lifecycle, and only forward transitions between stages are counted.
def run_bsa(texts): for t in texts: present = [code for code, pat in PATS if pat.search(t)] seq = sorted(set(present), key=lambda c: (STAGE[c], ...)) for a, b in zip(seq, seq[1:]): if STAGE[b] > STAGE[a]: # forward-only → event order F[a, b] += 1 # transition frequency matrix E = row_sums @ col_sums / total # expected under independence SR = (F - E) / sqrt(E) # standardised residuals
From the transition frequency matrix, expected counts under independence are computed from the row and column marginals, and per-cell standardised residuals measure how far each observed transition sits above chance. A global chi-square test confirms departure from independence. The strongest residuals are the reliable joints of the sequence, and they are what the state transition diagram renders, with arrow thickness banded to the residual.
The final product treats the corpus as a record of attention rather than of events. Three quantities are derived.
Volume is a direct aggregation. Spread and persistence require grouping articles that describe the same incident. Articles are clustered by a heuristic key combining the inferred state, a distinctive victim surname and the year, falling back to state and calendar week where no name is available. Clusters spanning more than 180 days are treated as recurring topics rather than single events and are dropped, so that a cluster approximates one incident's coverage arc.
# incident key: state + victim surname + year (fallback: state + ISO week) key = f"{state}|{surname}|{year}" if surname else f"{state}|wk{isoweek}" for incident, group in corpus.groupby(key): span = (group.date.max() - group.date.min()).days # persistence outlets = group.source.nunique() # spread volume = len(group) # volume
Once incidents carry coordinates, they can be read against the built environment. GRID3 Nigeria is the source for most layers — its data is published on ArcGIS (owner mlukang_GRID3) and reachable through the ArcGIS REST API — and the pipeline pulls health facilities (51,022 points), schools (107,670) and water points (28,967) in full, computing for each incident the great-circle distance to the nearest facility of each type. For police stations the choice of layer matters: GRID3's official layer (802 points) is administratively sparse and self-described as incomplete, whereas OpenStreetMap, though smaller in total, is denser in the mapped areas where incidents actually cluster and supplies a closer station for the majority of incidents. The main analysis therefore uses OpenStreetMap as an indicative primary for police and retains the GRID3 official figures as an addendum for transparency. Roads (6.3 million line features) and settlement extents (2.6 million polygons) are available from GRID3 but too large to embed; they suit a spatial join in desktop GIS.
# GRID3 point layers via ArcGIS REST; OSM (Overpass) as the police primary GRID3 = {"health": "a0ed9627…", "schools": "8aef2085…", "water": "1506e7ea…"} police_primary = fetch_police_osm() # indicative, denser near incidents police_official = fetch_grid3_points("93239bea…") # GRID3, for the addendum for layer, item_id in GRID3.items(): dt = nearest_infrastructure(dt, fetch_grid3_points(item_id), col=f"nearest_{layer}_km")
The headline result is a marked infrastructure gap. Social infrastructure is effectively ubiquitous: the median incident lies about half a kilometre from the nearest school and under a kilometre from the nearest health facility. Water points are sparser, and security infrastructure sparser still: the median incident sits over twenty kilometres from the nearest police station on the OpenStreetMap layer, only a minority fall within ten kilometres, and incidents that end in death sit farther from policing than those resolved with a release. The discriminating spatial variable is therefore distance to policing, and it is greatest in the northwest banditry belt.
Not all context is a distance. GRID3 also publishes a socioeconomic vulnerability score for each of Nigeria's 774 local government areas, as polygons carrying a five-band index. Rather than a nearest-point calculation, this is an attribute join: each incident is located within its LGA by point-in-polygon test and inherits that LGA's vulnerability band. The share of incidents ending in death can then be read across the bands. Almost every incident (1,139 of 1,140) matched an LGA, and lethality is lowest in the least-vulnerable band and elevated in the more-vulnerable bands, so vulnerability is associated with outcome as well as with occurrence.
from shapely.geometry import shape, Point from shapely.strtree import STRtree tree = STRtree(lga_polygons) # 774 LGAs, socio_index 1–5 for incident in dt: pt = Point(incident.lon, incident.lat) for idx in tree.query(pt): if lga_polygons[idx].contains(pt): incident.lga_vulnerability = socio_index[idx]; break
Two GRID3 layers are far too large to download whole — roads run to 6.3 million line features and settlement extents to 2.6 million polygons — but neither needs to be. The principle is to fetch only what the incidents require, and to ask the question that the data can actually answer.
For roads, distance to any road is meaningless because residential streets (3.8 million of them) put every point within metres of a road. The discriminating network is the highways, so the pipeline filters the layer by its class attribute to trunk and primary roads (about 94,000 segments), fetches only those, and measures each incident's distance to the nearest one with a local planar projection. The result is sharp: the median incident lies about a kilometre from a major road and roughly six in ten are within two kilometres, confirming that K4R is overwhelmingly a roadside phenomenon, with fatal incidents sitting slightly farther out.
# fetch only trunk/primary highways (~94k), not all 6.3M roads paths = fetch_grid3_lines(where="class='trunk' OR class='primary'") dt = nearest_line_km(dt, paths, out_col="nearest_major_road_km")
For settlement density, no geometry is downloaded at all. Each settlement block carries a building_count, and "how built-up is the area around an incident" is a single aggregate query: the sum of building counts within a five-kilometre buffer of the point. One cheap statistics request per incident (deduplicated by coordinate) returns the number, ranging here from a few thousand buildings in sparse rural areas to over two hundred thousand in dense cities, with lethality slightly elevated in small towns rather than in the largest cities.
# buffered aggregate — one statistics query per incident, no bulk download dt = settlement_density(dt, radius_m=5000, out_col="buildings_within_5km")
The same two problems can be solved in QGIS for those who prefer a graphical workflow. Add the GRID3 layers as ArcGIS REST connections; for roads, filter to trunk and primary with a layer query, then Vector → Analysis → Distance to Nearest Hub (or the NNJoin plugin) from the incident points; for settlement density, load the incident buffers and use Count Points/Sum in Polygon, or the Join Attributes by Location (Summary) tool with the building-count field. The logic is identical to the code above — filter or buffer first, so only the relevant local features are ever processed.
The whole procedure is provided as a single script, k4r_pipeline.py, with no dependencies beyond pandas, numpy, requests and openpyxl. To run it on a similar dataset, set the column names in the configuration block and call:
pip install pandas numpy requests openpyxl # structured extraction, BSA and longitudinal metrics (no network): python k4r_pipeline.py --input raw.xlsx --sheet Sheet1 --outdir ./out # add the geocoding and state-correction stage (uses OpenStreetMap): python k4r_pipeline.py --input raw.xlsx --outdir ./out --geocode # add spatial proximity to police stations (OSM/GRID3): python k4r_pipeline.py --input raw.xlsx --outdir ./out --geocode --proximity
The script writes the incident datatable, the behaviour-sequence results and the longitudinal metrics to the output directory, and caches geocoding results so that a re-run is fast and resumable. Each stage is a self-contained function, so individual stages can be imported and applied on their own.
Three limits bound every product of this pipeline. The corpus is media-visible, so cases that are never reported, including those resolved by quiet ransom payment, fall outside it; the outputs describe reported kidnapping rather than all kidnapping. Fields are read by textual pattern rather than coded by hand for each incident, which trades some per-case precision for coverage at scale, and the provenance flags record where that trade is most consequential. And the behavioural analysis is cross-sectional and lag-one: it captures which behaviour follows which, not duration, and not how patterns are shifting over time.
On ethics, all inputs are already-published news reports; no private or victim-identifying data is created beyond what the sources have made public, and inferred attributes such as gender are reported only as corpus-level distributions. Where a value cannot be supported from the text, it is left blank rather than estimated.