Lead optimization is where discovery programmes quietly go wrong. A hit binds well, the team commits, and eighteen months later a cardiac or hepatic liability that was visible in the starting structure ends the series. The liability was usually not hidden. It was simply not weighed against potency at the point where the series was chosen.

This case study runs that weighing in silico, three times over, against NEK2. NEK2 is a serine threonine kinase that regulates centrosome separation at the onset of mitosis and is overexpressed across several cancers, which makes it a target where ATP-competitive inhibitors are a reasonable ambition. We took a small compound panel, found a genuine hit, diagnosed why it would fail, and then asked a generative model to fix it. Each round fixed what it was asked to fix and quietly gave something back.

What these numbers are: Every affinity below is an AutoDock-Vina score, not a measured Ki, and Vina’s error on benchmark sets runs to roughly two to three kcal/mol. Differences smaller than that are not resolvable, so read the large shifts as signal and the fine rankings as ordering rather than measurement. Every toxicity value is an ADMET-AI prediction from human-trained models. Nothing here has been near a bench. 

Finding the hit

Eleven compounds were docked against the NEK2 ATP site using the 1.90 Å co-crystal structure in PDB 5M53, which captures human NEK2 bound to an arylaminopurine inhibitor and therefore presents a pocket already shaped to accept an ATP-competitive ligand. One detail of that choice matters downstream. The authors who deposited 5M53 reported it as the first Nek2 inhibitor complex in a DFG-in conformation, which means every affinity in this study is conditional on that particular pre-active state of the kinase rather than on NEK2 in general. All eleven compounds completed, none were skipped, and five poses were generated per ligand.

Table 1. Initial docking of the eleven-compound panel against NEK2 (5M53), ranked by best affinity. Lower is stronger.

Compound 3 came out cleanly ahead at −8.5 kcal/mol. Below it, four compounds cluster between −7.7 and −7.9, close enough together that ranking them against one another would be reading noise. The spread across the whole panel is 2.5 kcal/mol, which is wide enough that the top and bottom of the panel are genuinely different, even if neighbours within it are not. Compound 3 became the lead.

Diagnosing why the lead would fail

Running compound 3 through ADMET-AI produced the result that shaped the rest of the campaign. Predicted hERG liability came back at 0.858 and ClinTox at 0.730, both high. The compound also carried a dual CYP2D6 inhibitor and substrate profile, meaning it could accumulate and interfere with co-administered drugs at the same time, and its high predicted blood brain barrier penetration made the cardiac signal more concerning rather than less.

The liability was the molecule, not a substituent. Compound 3 has a cationic amphiphilic drug architecture. A flat quinoline, a bridged tertiary amine with a basic nitrogen around pKa 9, and a low TPSA of 36.4 Ų. That combination of a planar aromatic system and a protonatable basic centre in a lipophilic molecule is the textbook hERG pharmacophore. It is also, unfortunately, most of what was packing the hydrophobic ATP pocket and giving the compound its −8.5. Potency and liability were being carried by the same atoms.

This is the situation that makes lead optimization hard. When a liability sits on a peripheral group you trim the group. When it is distributed across the scaffold itself, as here, there is no trimming available and the core has to change. That is what happened across the three rounds below, and it is worth saying plainly at the outset that the molecules the campaign ended with share no scaffold with the molecule it started from. The quinoline is gone. The basic amine is gone. What was retained is the target, the pocket and the requirement to bind it.

Round one, aimed at hERG

REINVENT4 was run against compound 3 with the cationic amphiphilic features penalised and structural alerts filtered. It returned 98 valid molecules with a tightly converged composite score between 0.941 and 0.965, mean QED of 0.928, molecular weights between 256 and 357 Da, and no structural alerts firing anywhere in the set.

Table 2. Round one toxicity outcome against the parent compound.

On the axis it was asked to optimise, the round worked as well as anyone could reasonably ask. Mean predicted hERG fell from 0.858 to 0.260, with the best analogue at 0.004. Of the 98 molecules, 82 sat in the low-risk band below 0.5 and only one remained above 0.7. TPSA roughly doubled, which is the structural signature of the model having genuinely dismantled the cationic amphiphilic architecture rather than cosmetically decorating it.

Then the compounds went back into the pocket

The ten best-balanced analogues were re-docked against 5M53. This is where the round showed its price.

Table 3. Round one analogues re-docked against NEK2. Δ is the affinity lost relative to the parent’s −8.5 kcal/mol.

Every analogue lost binding. The best held at −7.7 kcal/mol, and the mean across the ten was −6.83, a shift of about 1.7 kcal/mol from the parent. That mean shift is large enough to sit outside docking noise and be believed. The individual gaps between neighbouring analogues are not. The direction is unambiguous even where the fine ordering is not, and the direction was down.

This was the expected tension rather than a surprise. The parent’s affinity was partly produced by the planar quinoline packing the hydrophobic pocket and the basic amine making its own contacts. Penalising those features removed both the liability and a portion of the binding energy, because they were the same chemistry.

And a second liability walked in behind the first. The best round-one analogue reached hERG 0.010 and ClinTox 0.039, with a clean AMES, no CYP inhibition and excellent predicted oral absorption. It also came back with a predicted DILI of 0.981 and a carcinogenicity flag at 0.597. The model had converged on diaryl sulfonyl and carboxylic acid warheads, which are exactly the groups that generate reactive metabolites capable of forming covalent adducts in hepatocytes, and on an aryl bromide that can be oxidised to a reactive arene oxide. Optimising one endpoint had steered the whole population into a different one. 

Round two, aimed at everything

The second REINVENT4 run added DILI and carcinogenicity penalties alongside the cationic amphiphilic constraints and returned 100 molecules. Widening the objective produced exactly the effect that widening an objective usually produces.

Table 4. Round two toxicity distribution across all 100 generated molecules.

The honest reading. The round did what it was asked on the new axes. Carcinogenicity came back clean in 93 of 100 molecules and AMES in 84, both substantial improvements over the sulfonyl and bromide-heavy chemistry of round one. But hERG partially returned. Mean predicted hERG rose from 0.260 to 0.560 and 39 molecules landed back in the high band. Spreading the reward across three toxicity axes meant no single axis received the near-total suppression that the hERG-only round achieved. The optimisation budget is finite, and every objective added to it is paid for by the others.

Ten molecules cleared all five gates simultaneously, requiring hERG below 0.5, DILI below 0.5, AMES below 0.5, carcinogenicity below 0.5 and ClinTox below 0.3. Those ten were re-docked.

Reading the final set properly

The table below merges the round two toxicity profile with the round two docking result, which is the only view that supports a decision. Sorted by affinity, it makes the campaign’s last trap immediately visible.

Table 5. The ten gate-passing molecules with docking and toxicity together, sorted by Vina affinity. The highlighted row is the compound we would advance, which is not the one that docked best.

Binding recovered. The strongest compound in the set reached −8.0 kcal/mol against the parent’s −8.5, and five of the ten sit at −7.4 or better. Given Vina’s error, the top six of this set are not meaningfully distinguishable from one another on affinity, and the honest statement is that round two recovered the binding that round one gave up while keeping the toxicity gates closed.

The compound we would advance

COc1ccc(C(=O)NCC(=O)N2CCCCC2)cc1 docks at −7.4 kcal/mol, which is within docking noise of the strongest compound in the set, and carries predicted hERG of 0.194 and DILI of 0.270. Against the top-docking molecule that is roughly forty percent lower on both of the endpoints that matter here, bought with a difference in affinity the method cannot resolve.

The structural argument is stronger than the numerical one. This molecule is a methoxybenzamide linked through a glycine to a piperidine amide. Every nitrogen in it is an amide nitrogen rather than a basic centre, and its SlogP is 1.44. Both halves of the cationic amphiphilic pharmacophore that condemned compound 3 are genuinely absent rather than merely predicted away, and it carries none of the sulfonyl or carboxylic acid warheads that drove the round one hepatotoxicity. It is a plain, unglamorous, synthesisable molecule with nothing obviously wrong with it, which at this stage of a programme is the most valuable thing a compound can be.

Two alternates are worth keeping. CCC1(c2ccccc2)N=C(N)N(C2CCCCC2)C1=O carries the lowest total predicted toxicity burden in the set with DILI at 0.087 and docks at −7.3, though its aminoimidazolone reintroduces a basic centre, which is precisely the feature the campaign spent round one removing and which the model does not flag. That divergence between prediction and medicinal chemistry intuition is worth resolving before it is advanced. Cn1nnc2c1C(C(=O)NC1(c3ccccc3)CC1)SCC2 has the lowest predicted hERG in the entire campaign at 0.025 and the best drug-likeness at QED 0.941, with a borderline DILI of 0.410 as its only mark.

What the platform contributed

Three models, three rounds, one workflow. AutoDock Vina, ADMET-AI and REINVENT4 ran as a single orchestrated loop, with the toxicity profile of each round feeding directly back into the generator’s constraints for the next. The structural alerts derived from fragmenting the parent were encoded and applied without leaving the workflow.

The full endpoint panel, every round. The round one hepatotoxicity problem was visible only because every endpoint was reported rather than collapsed into a composite. On the composite alone, the round one winner looked like a triumph. Its DILI of 0.981 is what actually mattered, and a workflow that reports one number would have advanced it.

Docking and toxicity read together. Merging the final docking and toxicity tables is what exposed that the best-binding gate-passer was the worst gate-passer on both target endpoints. Held in separate tables, as they were generated, that compound reads as the obvious lead.

Where this goes next

The most valuable change to this workflow would be to stop optimising sequentially. Every round here generated molecules, filtered them on toxicity, took the top ten and only then docked them, which means compounds with excellent binding and merely acceptable toxicity were never docked at all. The true Pareto front between affinity and safety was never observed, only a toxicity-selected slice of it. Scoring docking and toxicity inside a single objective function would let the search find the region where both hold, rather than discovering the trade-off after the fact in three separate instalments.

Beyond that, the immediate steps are ordinary. Confirm the binding mode of the advanced compound with a three-dimensional interaction fingerprint against the 5M53 pocket, so that hinge engagement is demonstrated rather than assumed. Establish a synthetic route and confirm accessibility. Then a CRO handoff for a NEK2 kinase domain IC50 and a hepatocyte toxicity panel, because measurement is the only thing that converts any of this into a result.

A fair word on scope. This campaign was in silico from end to end. Every affinity is a docking score with a two to three kcal/mol error bar, which means the fine rankings within each round order the compounds without measuring them. Every toxicity number is a prediction from human-trained models applied to scaffolds that in several cases sit well away from their training chemistry, and predictions on novel chemotypes deserve more scepticism than predictions on familiar ones. All docking was performed against a single rigid receptor conformation, the DFG-in state captured in 5M53, so the affinities describe complementarity to that conformer rather than to the conformational ensemble the kinase actually samples. The optimisation replaced the scaffold entirely rather than modifying it, so no structure activity relationship was carried forward from the parent and the final molecules must be treated as new chemical matter with their own unexplored liabilities. Only ten molecules from each round of roughly a hundred were re-docked, so the reported affinity distributions describe a toxicity-selected subset rather than the generated population. What the campaign does show cleanly is that the workflow found a real hit, correctly identified why it would fail, quantified what removing that failure cost in binding, caught a second liability it introduced along the way, and ended with a compound that holds affinity inside the noise of the best binder while cutting both target liabilities by roughly half. That last comparison is the result, and it only exists because the docking and toxicity tables were read together.