VM Instrumentation Without an Agent

The Java Instrumentation API is typically accessed through the -javaagent command line flag, but what if you want instrumentation capabilities in an already running process? Today I'll show you how to create a fully functional Instrumentation object without ever using -javaagent.

The 'What' and 'Why' of Instrumentation

Before we dive into the native-side "how," let's quickly cover the "what" and "why."

So, what even is instrumentation? At its core, the java.lang.instrument API is a way to hook into the JVM's class-loading process. It lets you get the raw bytecode of a class before it's used, and you can change it. This is called "transforming." You can add logging, check for security stuff, or monitor performance, all without touching the original .java file.

This Java API is just a nice wrapper around a deeper, native layer. This is the JVMTI (JVM Tool Interface). To do the really cool stuff, like redefining classes that are already loaded, the native code needs to ask the JVM for permission. These permissions are called capabilities, like can_redefine_classes or can_retransform_classes.

This brings us to the whole point of this post. The standard -javaagent flag is great, but you have to specify it on the command line before the process starts. What if you want to attach a profiler or a security tool to a Java server that's already running and you can't restart it? This technique shows you how to bypass the normal startup requirements, build all the internal JVM structures by hand, and get a fully-powered Instrumentation object in a live process.

The Standard Approach

Normally, you'd get instrumentation like this

// Agent.java
public class Agent {
    public static void premain(String args, Instrumentation inst) {
        // inst is provided by the JVM
        inst.addTransformer(new MyTransformer());
    }
}

Then run with

java -javaagent:agent.jar -jar application.jar

The JVM creates an Instrumentation instance and passes it to your premain method, but what actually happens under the hood?

JPLIS Internals

The Java programming language Instrumentation services (JPLIS) is the native component that implements the Instrumentation API. When you use -javaagent, the JVM loads your agent jar and creates a native JPLISAgent structure.

This structure lives in JPLISAgent.h and looks like this

// from JPLISAgent.h
struct _JPLISAgent {
    JavaVM* mJVM;
    JPLISEnvironment mNormalEnvironment;
    JPLISEnvironment mRetransformEnvironment;
    jobject mInstrumentationImpl; // handle to the Instrumentation instance
    jmethodID mPremainCaller;
    jmethodID mAgentmainCaller;
    jmethodID mTransform;
    jboolean mRedefineAdded;
    jboolean mNativeMethodPrefixAdded;
    // ... more fields
};

The key insight is that sun.instrument.InstrumentationImpl (the concrete implementation of the Instrumentation interface) is just a regular Java class that wraps this native structure. In the Java class, this is stored in a long field

// from sun.instrument.InstrumentationImpl.java
// needs to store a native pointer, so use 64 bits
private final     long            mNativeAgent;

Its constructor signature is

// from sun.instrument.InstrumentationImpl.java
private
InstrumentationImpl(long   nativeAgent,
                    boolean environmentSupportsRedefineClasses,
                    boolean environmentSupportsNativeMethodPrefix,
                    boolean printWarning) {
    mNativeAgent = nativeAgent;
    // ...
}

The first parameter is a pointer to the native JPLISAgent structure, cast to a long. The JVM doesn't actually validate that this pointer came from a legitimate agent loading process. it just stores it and uses it when you call Instrumentation methods.

Getting JVMTI Without an Agent

Before we can create a fake agent, we need JVMTI access. Normally this requires loading a native agent with -agentlib, but there's a simpler way.

From any native code running in the process, you can get the JVM handle

JavaVM* jvm = nullptr;
jsize vmCount = 0;
JNI_GetCreatedJavaVMs(&jvm, 1, &vmCount);

JNI_GetCreatedJavaVMs is part of the JNI Invocation API and returns handles to all JVMs in the current process. Once you have the JavaVM*, getting JVMTI is trivial

jvmtiEnv* jvmti = nullptr;
jvm->GetEnv((void**)&jvmti, JVMTI_VERSION_1_2);

Now you have full JVMTI access without any agent setup. But there's a catch, many JVMTI capabilities (like can_retransform_classes) can only be added during the OnLoad phase, which happens before the JVM fully starts up. If you try adding them later, you'll get JVMTI_ERROR_NOT_AVAILABLE.

This is where the fake agent trick comes in.

Creating a Fake JPLISAgent

The trick is to manually construct a JPLISAgent structure that looks exactly like one the JVM would have created during proper agent loading. First, we need to replicate the structure layout from JPLISAgent.h

struct _JPLISEnvironment {
    jvmtiEnv* mJVMTIEnv;
    JPLISAgent* mAgent;
    jboolean mIsRetransformer;
};

// this is the full struct from JPLISAgent.h
struct _JPLISAgent {
    JavaVM* mJVM;
    JPLISEnvironment mNormalEnvironment;
    JPLISEnvironment mRetransformEnvironment;
    jobject mInstrumentationImpl;
    jmethodID mPremainCaller;
    jmethodID mAgentmainCaller;
    jmethodID mTransform;
    jboolean mRedefineAvailable;
    jboolean mRedefineAdded;
    jboolean mNativeMethodPrefixAvailable;
    jboolean mNativeMethodPrefixAdded;
    char const* mAgentClassName;
    char const* mOptionsString;
    const char* mJarfile;
    jboolean mPrintWarning;
};

These structures must match OpenJDK's internal layout exactly, as the JVM will dereference this pointer when you use instrumentation methods. Any misalignment will cause crashes.

Now we create and initialise the fake agent

JPLISAgent* createDummyAgent(JavaVM* vm, jvmtiEnv* jvmti) {
    // allocate using JVMTI so it's in the right memory space
    JPLISAgent* agent = (JPLISAgent*)allocate(jvmti, sizeof(JPLISAgent));
    
    agent->mJVM = vm;
    agent->mNormalEnvironment.mJVMTIEnv = jvmti;
    agent->mNormalEnvironment.mAgent = agent;
    agent->mNormalEnvironment.mIsRetransformer = JNI_FALSE;
    
    // critical: leave this null
    agent->mRetransformEnvironment.mJVMTIEnv = nullptr;
    agent->mRetransformEnvironment.mAgent = agent;
    agent->mRetransformEnvironment.mIsRetransformer = JNI_TRUE;
    
    agent->mRedefineAdded = JNI_TRUE;
    agent->mNativeMethodPrefixAdded = JNI_TRUE;
    // ... initialise other fields
    
    // store in JVMTI local storage so callbacks can find it
    jvmti->SetEnvironmentLocalStorage(&agent->mNormalEnvironment);
    
    return agent;
}

The SetEnvironmentLocalStorage call is crucial. When JVMTI callbacks fire (like ClassFileLoadHook), the VM retrieves the agent structure using GetEnvironmentLocalStorage. If it's not set, callbacks won't work properly. This is exactly what the real initializeJPLISAgent in JPLISAgent.c does

// from JPLISAgent.c
JPLISInitializationError
initializeJPLISAgent(   JPLISAgent * agent,
                        JavaVM * vm,
                        jvmtiEnv * jvmtienv,
                        /*...*/) {
    // ...
    /* make sure we can recover either handle in either direction.
    * the agent has a ref to the jvmti; make it mutual
    */
    jvmtierror = (*jvmtienv)->SetEnvironmentLocalStorage(
                                            jvmtienv,
                                            &(agent->mNormalEnvironment));
    /* can be called from any phase */
    jplis_assert(jvmtierror == JVMTI_ERROR_NONE);
    // ...
}

We're just replicating the official setup.

There's a subtle trick with the retransform environment. By setting mRetransformEnvironment.mJVMTIEnv = nullptr but mNormalEnvironment.mJVMTIEnv = jvmti, we force the JVM to register the class file load hook in the normal environment. this is handled by the retransformableEnvironment function in JPLISAgent.c, which checks if the retransform environment already exists before creating a new one

// from JPLISAgent.c
jvmtiEnv *
retransformableEnvironment(JPLISAgent * agent) {
    jvmtiEnv * retransformerEnv       = NULL;
    // ...

    if (agent->mRetransformEnvironment.mJVMTIEnv != NULL) {
        return agent->mRetransformEnvironment.mJVMTIEnv;
    }
    jnierror = (*agent->mJVM)->GetEnv(  agent->mJVM,
                                    (void **) &retransformerEnv,
                                    JVMTI_VERSION_1_1);
    // ...
    // ... add can_retransform_classes capability ...
    // ...
    jvmtierror = (*retransformerEnv)->AddCapabilities(retransformerEnv, &desiredCapabilities);
    // ...
    // install the retransforming environment
    agent->mRetransformEnvironment.mJVMTIEnv = retransformerEnv;
    agent->mRetransformEnvironment.mIsRetransformer = JNI_TRUE;
    // ...
    return retransformerEnv;
}

By leaving our mRetransformEnvironment.mJVMTIEnv as NULL, we trick the instrumentation layer into using the normal environment for retransformation, which is what we want.

Instantiating InstrumentationImpl

Now comes the clever bit, directly instantiating sun.instrument.InstrumentationImpl via JNI.

jobject createInstrumentation(JavaVM* vm, JNIEnv* env, jvmtiEnv* jvmti) {
    JPLISAgent* agent = createDummyAgent(vm, jvmti);
    
    jclass instClass = env->FindClass("sun/instrument/InstrumentationImpl");
    jmethodID constructor = env->GetMethodID(
        instClass,
        "<init>",
        "(JZZZ)V"  // (long agentPtr, boolean, boolean, boolean)
    );
    
    jlong agentPtr = (jlong)(intptr_t)agent;
    jobject inst = env->NewObject(
        instClass,
        constructor,
        agentPtr,
        agent->mRedefineAdded,       // boolean
        agent->mNativeMethodPrefixAdded, // boolean
        JNI_FALSE                  // boolean (printWarning)
    );
    
    // make it global so it doesn't get GC'd
    inst = env->NewGlobalRef(inst);
    
    // cache the transform method
    jmethodID transformMethod = env->GetMethodID(instClass, 
        "transform",
        "(Ljava/lang/Module;Ljava/lang/ClassLoader;Ljava/lang/String;"
        "Ljava/lang/Class;Ljava/security/ProtectionDomain;[BZ)[B"
    );
    
    agent->mInstrumentationImpl = inst; // IMPORTANT: store the java object back into the native struct
    agent->mTransform = transformMethod;
    
    return inst;
}

This directly mimics the VM's own createInstrumentationImpl function in JPLISAgent.c, which is called during the VMInit phase

// from JPLISAgent.c
jboolean
createInstrumentationImpl( JNIEnv * jnienv,
                           JPLISAgent * agent) {
    // ...
    implClass = (*jnienv)->FindClass(   jnienv,
                                        JPLIS_INSTRUMENTIMPL_CLASSNAME);
    // ...
    constructorID = (*jnienv)->GetMethodID( jnienv,
                                            implClass,
                                            JPLIS_INSTRUMENTIMPL_CONSTRUCTOR_METHODNAME,
                                            JPLIS_INSTRUMENTIMPL_CONSTRUCTOR_METHODSIGNATURE);
    // ...
    jlong   peerReferenceAsScalar = (jlong)(intptr_t) agent;
    localReference = (*jnienv)->NewObject(  jnienv,
                                            implClass,
                                            constructorID,
                                            peerReferenceAsScalar,
                                            agent->mRedefineAdded,
                                            agent->mNativeMethodPrefixAdded,
                                            agent->mPrintWarning);
    // ...
    resultImpl = (*jnienv)->NewGlobalRef(jnienv, localReference);
    // ...
    agent->mInstrumentationImpl = resultImpl;
    agent->mTransform           = transformMethodID;
    // ...
    return !errorOutstanding;
}

As you can see, our "trick" is just a recreation of the VM's own setup process. The JVM doesn't validate this pointer at construction time, it just stores it as a field.

Adding Capabilities

There's still one problem, we need to add JVMTI capabilities, but we're past the OnLoad phase. Surprisingly, this actually works

jvmtiCapabilities caps = {};
caps.can_redefine_classes = 1;
caps.can_retransform_classes = 1;
caps.can_retransform_any_class = 1;
caps.can_set_native_method_prefix = 1;

jvmti->AddCapabilities(&caps);

So, while the documentation says these capabilities should only be added during OnLoad, in practice OpenJDK allows them to be added later. The VM just won't retroactively affect classes which have already been loaded. But for new classes and explicitly retransformed classes, it actually works fine.

This is, again, exactly how the agent itself adds capabilities when they're requested by Instrumentation methods

// from JPLISAgent.c
void
addRedefineClassesCapability(JPLISAgent * agent) {
    jvmtiEnv * jvmtienv = jvmti(agent);
    jvmtiCapabilities desiredCapabilities;
    jvmtiError        jvmtierror;

    if (agent->mRedefineAvailable && !agent->mRedefineAdded) {
        jvmtierror = (*jvmtienv)->GetCapabilities(jvmtienv, &desiredCapabilities);
        /* can be called from any phase */
        jplis_assert(jvmtierror == JVMTI_ERROR_NONE);
        desiredCapabilities.can_redefine_classes = 1;
        jvmtierror = (*jvmtienv)->AddCapabilities(jvmtienv, &desiredCapabilities);
        check_phase_ret(jvmtierror);
        
        // ...
        if (jvmtierror == JVMTI_ERROR_NONE) {
            agent->mRedefineAdded = JNI_TRUE;
        }
    }
}

The agent adds capabilities lazily, so we can too.

Conclusion

The Java instrumentation API seems like it requires -javaagent, but that's just the standard entry point. By understanding JPLIS internals and carefully constructing the native structures the JVM expects, we can create a fully functional Instrumentation object from scratch.

Thank you for reading <3