LA is a data-driven approach. The “what?” dimension refers to the kind of data used in the LA task. It also refers to the environments and contexts in which learning occurs. An interesting question in LA is where the educational data comes from. LA approaches use varied sources of educational data. These sources fall into two big categories: centralized educational systems and distributed learning environments. Centralized educational systems are well represented by learning management systems (LMS). LMS accumulate large logs of data of the students’ activities and interaction data, such as reading, writing, accessing and uploading learning material, taking tests, and sometimes have simple, built-in reporting tools (Romero and Ventura, 2007). LMS are often used in formal learning settings to enhance traditional face-to-face teaching methods or to support distance learning.

User-generated content, facilitated by ubiquitous technologies of production and cheap tools of creativity, has led to a vast amount of data produced by learners across several learning environments and systems. With the growth of user-generated content, LA based on data from distributed sources is becoming increasingly important and popular. Open and distributed learning environments are well represented by the personal learning environment (PLE) concept (Chatti et al., 2012b). PLEs compile data from a wide variety of sources beyond the LMS. The data comes from formal as well as informal learning channels. It can also come in different formats, distributed across space, time, and media.

The application of LA can be oriented toward different stakeholders, including students, teachers, (intelligent) tutors/mentors, educational institutions (administrators and faculty decision makers), researchers, and system designers with different perspectives, goals, and expectations from the LA exercise. Students will probably be interested in how analytics might improve their grades or help them build their personal learning environments. Teachers might be interested in how analytics can augment the effectiveness of their teaching practices or support them in adapting their teaching offerings to the needs of students. Educational institutions can use analytics tools to support decision making, identify potential students “at risk”, improve student success (i.e. student retention and graduation rates) (Campbell et al., 2007), develop student recruitment policies, adjust course planning, determine hiring needs, or make financial decisions (Educause, 2010).

There are many objectives in LA according to the particular point of view of the different stakeholders. Possible objectives of LA include monitoring, analysis, prediction, intervention, tutoring/mentoring, assessment, feedback, adaptation, personalization, recommendation, awareness, and reflection.

LA applies different methods and techniques to detect interesting patterns hidden in educational data sets. In this section, we describe four techniques that have received particular attention in the LA literature in the last couple of years, namely statistics, information visualization (IV), data mining (DM), and social network analysis (SNA).

The abundance of educational data, as pointed out in the “what?” dimension of the LA reference model, and the recent attention on the potentiality of efficient infrastructures for capturing and processing large amounts of data, known as big data, have resulted in a growing interest in big learning analytics among LA researchers and practitioners (Dawson et al., 2014). Big learning analytics refers to leveraging big data analytics methods to generate value in TEL environments.

Harnessing big data in the TEL domain has enormous potential. LA stakeholders have access to a massive volume of data from learners’ activities across various learning environments which, through the use of big data analytics methods, can be used to develop a greater understanding of the learning experiences and processes in the new open, networked, and increasingly complex learning environments.

A key challenge in big learning analytics is how to aggregate and integrate raw data from multiple, heterogeneous sources, often available in different formats, to create a useful educational data set that reflects the distributed activities of the learner; thus leading to more precise and solid LA results. Furthermore, handling of big data is a technical challenge because efficient analytics methods and tools have to be implemented to deliver meaningful results without too much delay, so that stakeholders have the opportunity to act on newly gained information in time. Strategies and best practices on harnessing big data in TEL have to be found and shared by the LA research community.

As stressed in the “what?” dimension of the LA reference model, learning is increasingly taking place in open and networked learning environments. In the past few years, Massive Open Online Courses (MOOCs) have gained popularity as a new form of open learning. MOOCs can be roughly classified in two groups. On the one hand there are xMOOCs. Although they gained a lot of attention they can be seen as a replication of traditional learning management systems (LMS) at a larger scale. Still they are closed, centralized, structured, and teacher-centered courses that emphasize video lectures and assignments. In xMOOCs all services available are predetermined and offered within the platform itself. On the other hand there is the contrasting idea of cMOOCs combining MOOCs with the concept of personal learning environment (PLE). In contrast to xMOOCs, cMOOCs are open-ended, distributed, networked, and learner-directed learning environments where the learning services are not predetermined, and most activities take place outside the platform (Yousef et al., 2014a).

MOOC environments provide an exciting opportunity for the LA community. They capture and store large data sets from learners’ activities that can provide insight into the learning processes. MOOCs house a wide range of participants with diverse interests and needs. Understanding how learners are engaging with the MOOC and analyzing their activities in the course is a first step towards more effective MOOCs. Applying LA in MOOCs can thus provide course providers and platform developers with insights for designing effective learning environments that best meet the needs of MOOC participants.

In addition, due to the open nature of MOOCs, learners are often overwhelmed with the flow of information in MOOCs. Moreover, in MOOCs it is difficult to provide personal feedback to a massive number of learners. LA can provide great support to learners in their MOOC experience. LA that focuses on the perspectives of learners can help to form the basis for increased learner awareness and effective personalized learning.

Recognizing these opportunities, several MOOC studies recommended to apply LA tools to monitor the learning process, identify difficulties, discover learning patterns, provide feedback, and support learners in reflecting on their own learning experience (Fournier et al., 2011; Yousef et al., 2014b). Despite the wide agreement that LA can provide value to different MOOC stakeholders, the application of LA techniques on MOOC datasets is rather limited until now. Most of the LA implementations in MOOCs so far meet the needs of the course providers and have primarily focused on addressing low completion rates, investigating learning patterns, and supporting intervention (Clow, 2013b; Mirriahi & Dawson, 2013). A focus on the perspectives of learners in MOOCs has the potential to extend LA beyond completion rates and interventions to support personalization, feedback, assessment, recommendation, awareness, and self-reflection.

As compared to xMOOCs, cMOOCs offer particular challenges for LA practice. In fact, in a cMOOC, participation is emergent, fragmented, diffuse, and diverse (Clow, 2013b; McAulay et al., 2010) and most of the learning activities happen outside the platform across different learning environments. Operating in cMOOCs require a shift towards LA on more challenging data sets across a variety of different sites with different standards, owners, and levels of access (Ferguson, 2012; Fournier et al., 2011).

Another important research direction related to the “what?” dimension of the LA reference model is mobile learning analytics. The widespread use of mobile technologies has led to an increasing interest in mobile learning (Thüs et al., 2012). Mobile technologies open an unprecedented opportunity to conduct research on mobile learning analytics. Mobile learning analytics refers to LA mediated through mobile and ubiquitous technologies. It includes harnessing the data of mobile learners and presenting the analytics results on mobile devices. Mobile learning analytics bears a number of challenges. It has to handle the specificity of the mobile devices with respect to their constraints. A mobile device has only a specific amount of battery power which discharges very quickly when network communication is extensively used. The limited bandwidth is also one gap which has to be considered seriously, as huge amounts of data required for a LA task cannot be transferred so easily. Data has to be transferred in real time as the usage intervals of the mobile learning platform are frequent but very short, thus the user cannot be kept waiting for a long period of time. Furthermore, a mobile device is not as powerful as a personal computer. Therefore, time-consuming computing operations have to be done on external servers or by other technologies which provide enough computation power to solve hard computational tasks. The mobile device should only process the presentation of the analytics results. The possibility to display graphical elements (e.g. dashboards) on the screen depends on its resolution and size. Furthermore, mobile learning is normally done on-the-fly, this means it is very fast and mostly during spare time. Thus, the provided analytics results have to be personalized to the needs of the user. It is also important to leverage the context attribute in mobile learning in order to give learners the support they need when, how, and where they need it.

In addition to data and environments, the “what?” dimension of the LA reference model refers to the context in which learning occurs. In fact, learning is happening in a world without boundaries. Learners are collaborating more than ever beyond classroom boundaries which become more and more irrelevant. Thus, for data gathering, the context in which events take place plays a pivotal role.

Context is crucial in mobile learning environments. In fact, a major benefit of mobile devices is that they enable learning across contexts. Harnessing context in a mobile learning experience has a wide range of benefits including personalization, adaptation, intelligent feedback, and recommendation. Adding context information to a set of mobile applications may lead to presenting more personalized data to the learner and to more motivating mobile learning experiences.

Context is also a key factor in workplace learning, self-directed learning, and lifelong learning to achieve effective learning activities. Nowadays, knowledge workers and self-directed lifelong learners perform in highly complex knowledge environments. They have to deal with a wide range of activities they have to manage every day. And, they have to combine learning activities and their private and professional daily life. The challenge here is the linking of formal and informal learning activities and the bridging of learning activities across different contexts.

To date, there is little support for learners that typically try to learn in different contexts and must align their learning activities to their daily routines. LA tools that are smoothly integrated into learners’ daily life can offer solutions to address this challenge. There is a need to focus on the context attribute in LA by developing methods for effective context capturing and modeling. A context model should reflect a complete picture of the learner’s context information.

As stated in the “who?” dimension of the LA reference model, the application of LA can be oriented toward different stakeholders. In the internet era, keeping personal information private is a big challenge. This challenge transfers to LA as well. There are different approaches and research is being done in order to understand and tackle the ethical and privacy issues that arise with practical use of LA in the student context. Pardo and Siemens (2014), for instance, provide four practical principles that researchers should consider about privacy when working on an LA tool. These practical principles are transparency, student control over the data, security, and accountability and assessment. Slade and Prinsloo (2013) propose an ethical framework with six guiding principles for privacy-aware LA implementations. These include (1) learning analytics as moral practice, (2) students as agents, (3) student identity and performance are temporal dynamic constructs, (4) student success is a complex and multidimensional phenomenon, (5) transparency, and (6) higher education cannot afford to not use data. Some of the guiding principles are overlapping, or the same as the principles suggested by Pardo and Siemens (2014). It is crucial to embrace these principles while creating new opportunities for analytics-supported learning.

In general, we have a vast range of possibilities for LA research, but we should always keep in hindsight the ethical and privacy challenges of the stakeholders while trying to institutionalize LA and include it into the day-to-day learning activities. Further research is needed to investigate approaches that can help design LA solutions where ethical and privacy issues are considered.

In relation to the “why?” dimension in the LA reference model, the various objectives in LA (e.g. monitoring, analysis, prediction, intervention, tutoring, mentoring, assessment, feedback, adaptation, personalization, recommendation, awareness, reflection) need a tailored set of indicators and metrics to serve different stakeholders with very diverse needs. Current implementations of LA rely on a predefined set of indicators and metrics. This raises questions about what sort of indicators and metrics are helpful in each learning situation. Ideally, LA tools should support an exploratory, real-time user experience that enables data exploration and visualization manipulation based on individual interests of users. The challenge is thus to define the right Goal / Indicator / Metric (GIM) triple before starting the LA exercise. Giving users the opportunity to interact with the LA tool by defining their goal and specifying the indicator/metric to be applied is a crucial step for more effective analytics results. This would also make the analytics process more transparent, enabling users to see what kind of data is being used and for which purpose. From a technical perspective, templates and rule engines can be used to achieve goal-oriented LA for improved personalized learning.

The “why?” dimension of the LA reference model in particular encompasses learner-centered LA objectives, such as intelligent feedback, adaptation, personalization, and recommendation. In order to achieve these objectives, learner modeling is a crucial task. A learner (user) model is a representation of information about an individual learner that is essential for adaptation and personalization tasks. The six most popular and useful features in learner modeling include the learner’s knowledge, interests, goals, background, individual traits, and context (Brusilovsky and Millan, 2007).The challenge is to create a thorough “polycontextual” learner model that can be used to trigger effective intervention, adaptation, personalization, or recommendation actions. This is a highly challenging task since learner activities are often distributed over open and increasingly complex learning environments. A big challenge to tackle here is lifelong learner modeling. Kay and Kummerfeld (2011) define a lifelong learner model as a store for the collection of learning data about an individual learner. The authors note that to be useful, a lifelong learner model should be able to hold many forms of learning data from diverse sources and to make that information available in a suitable form to support learning. The aim is that data gathered from different learning environments would be fed into a personal lifelong learner model, which would be the location where the learner can archive all learning activities throughout her life. This model must store very heterogeneous data. It would also contain both long-term and short-term goals of the learner.

The capacity to build a detailed picture of the learner across a broader learning context beyond the classroom would provide more accurate analytics results. Future research in this area should focus on effective and efficient LA methods to mine, aggregate, manage, share, reuse, visualize, interpret, and leverage learner models in order to better induce and support personalized and lifelong learning. Thereby, several issues have to be taken into account, including questions about privacy, scalability, integration, and interoperability.

In relation to the “why?” dimension in the LA reference model, assessment constitutes a core component of the learning process and one important objective for LA. Recent years have seen a growing interest in learner-centered, open, and networked learning models (e.g. PLEs, OER, MOOCs). Unlike assessment in classroom settings, the methodology to assess learning in open environments has not yet been well investigated. The move to open learning models raises challenges with regard to the assessment of learning. Learner-centered and networked learning requires new assessment models that address how to recognize and evaluate self-directed, network, and lifelong learning achievements. We need thus to think about an innovative open assessment framework for accrediting and recognizing open learning in an efficient, valid, and reliable way.

Open assessment is an all-encompassing term combining different assessment methods for recognizing learning in open and networked learning environments. In open assessment, summative, formative, formal, informal, peer, network, self-, and e-assessment converge to allow lifelong learners to gain recognition of their learning. It is an agile way of assessment where anyone, anytime, anywhere, can participate towards the assessment goal. Open assessment is an ongoing process across time, locations, and devices where everyone can be assessor and assessee.

Open assessment is a major goal for LA. However, harnessing LA for assessment purposes has not yet been well investigated. It is important to implement LA-based open assessment methods throughout the entire learning process that enable learners to continuously receive feedback, validation, and recognition of their learning, thus motivating and engaging them in their learning journey. LA can support open assessment in various ways. These include:

The “how?” dimension of the LA reference model suggests different techniques and tools that can be applied in LA. Integration is a challenge that has to be taken into account in the development of LA tools. In fact, LA is most effective when it is an integrated part of the learning environment. Hence, integration of LA into the learning practice of the different stakeholders is important and should be accompanied by appropriate didactical and organizational frameworks. LA developers will have to provide tools, which should optimally be embedded into the standard TEL toolsets of the learners and teachers.

Moreover, effective LA tools are those, which minimize the time frame between analysis and action. Thus, it is crucial to follow an embedded LA approach by implementing contextualized LA applications that give useful information at the right place and time; thus getting away from traditional dashboard-based LA.

Learning analytics design patterns constitute a further research direction related the “how?” dimension of the LA reference model. For the development of usable and useful LA tools, guidelines and design patterns should be taken into account. Appropriate visualizations could make a significant contribution to understanding the large amounts of educational data. Statistical, filtering, and mining tools should be designed in a way that can help learners, teachers, and institutions to achieve their analytics objectives without the need for having an extensive knowledge of the techniques underlying these tools. In particular, educational data mining tools should be designed for non-expert users in data mining. Moreover, performance, scalability, and extensibility should be taken into consideration in order to allow for incremental extension of data volume and analytics functionality after a system has been deployed. Also, student and teacher involvement is the key to a wider user acceptance, which is required if LA tools are to serve the intended objective of improving learning and teaching. The number and hierarchy of stakeholders bear conflicts. Involving all stakeholders and supporting all their interests is a complex task that has to be solved, since some of these interests might be contradictory. For instance, the usage of LA by administration staff for finding best practice examples of technology-enhanced teaching could offend teachers because they might feel assessed and controlled. The same could be true for students who might fear that personal data will not be used for formative evaluation but for assessment and grading. This could lead to the unintended effect that teachers and/or students are not motivated to use new technologies and participate in analytics-based TEL scenarios.

Another research opportunity related the “how?” dimension of the LA reference model is the effective evaluation of the developed LA tools. LA tools (as most interfaces) are generative artifacts: They do not have value in themselves, but they produce results in a particular context. This means that an LA tool is used by a particular user (e.g. student, teacher, mentor, researcher), on a particular data set, and for a particular reason. Hence, the evaluation of such tool is very complicated and diverse. Ellis and Dix (2006) argue that for search of validation of generative artefacts, empirical evaluation is methodologically unsound. Any empirical evaluation, cannot tell you, in itself, that the LA tool works, or does not work. Dyckhoff et al. (2013) summarize the goals of LA evaluation in three major goal groups: (a) explicitly inform the design of LA tools, (b) involve a behavioral reaction of the teacher, and (c) involve a behavioral reaction of the student. The authors inquire how does an LA tool, which has been carefully designed according to the mentioned goals and requirements, influence the behavior of its users, and therefore how does this tool influence practical learning situations? In their research they found only few publications that reported findings related to the behavioral reaction of teachers and students while using LA tools. They strongly call for impact analysis and usefulness evaluation of the LA tools.

Effective evaluation is difficult and problem-prone, but it is essential to support the LA task. There are several aspects that one needs to take into account in order to do an effective evaluation. The first thing one has to consider is the purpose, and the gains from the evaluation. The evaluators need to carefully design the goals and try to meet them with the tool’s evaluation. Once the evaluators have set their goals for the evaluation, they need to think about the measures and tasks that are going to be encompassed in the evaluation. It is important to define the appropriate indicators and metrics. The evaluators also need to consider collecting qualitative data and use qualitative methods in pair with the quantitative evaluation (wherever possible). A mixed-method evaluation approach that combines both quantitative and qualitative evaluation methods can be very powerful in order to capture when, how often, and why a peculiar behavior happens in the learning environment.

How can we evaluate the usability, usefulness, and learning impact of LA? While usability is relatively easy to evaluate, the challenge is to investigate how LA could impact learning and how this could be evaluated. Measuring the impact of LA tools is a challenging task, as the process needs long periods of time as well as a lot of effort and active participation from researchers and participants. Moreover, the analysis of the qualitative data from the evaluation is always prone to personal interpretations and biased while making conclusions (Dyckhoff et al., 2013). Thus, further research is required to investigate effective mixed-method evaluation approaches that do not only focus on the usability of the LA tool, but also aim at measuring its impact on learning.